Logical router with multiple routing components

ABSTRACT

Some embodiments provide a method for handling failure at one of several peer centralized components of a logical router. At a first one of the peer centralized components of the logical router, the method detects that a second one of the peer centralized components has failed. In response to the detection, the method automatically identifies a network layer address of the failed second peer. The method assumes responsibility for data traffic to the failed peer by broadcasting a message on a logical switch that connects all of the peer centralized components and a distributed component of the logical router. The message instructs recipients to associate the identified network layer address with a data link layer address of the first peer centralized component.

BACKGROUND

Typical physical networks contain several physical routers to perform L3forwarding (i.e., routing). When a first machine wants to send a packetto a second machine located on a different IP subnet, the packet is sentto a router that uses a destination IP address of the packet todetermine through which of its physical interfaces the packet should besent. Larger networks will contain multiple routers, such that if one ofthe routers fails, the packets can be routed along a different pathbetween the first machine and the second machine.

In logical networks, user-defined data compute nodes (e.g., virtualmachines) on different subnets may need to communicate with each otheras well. In this case, tenants may define a network for virtualizationthat includes both logical switches and logical routers. Methods forimplementing the logical routers to adequately serve such virtualizedlogical networks in datacenters are needed.

BRIEF SUMMARY

Some embodiments provide a method for implementing a logical router in anetwork (e.g., in a datacenter). In some embodiments, the method isperformed by a management plane that centrally manages the network(e.g., implemented in a network controller). The method, in someembodiments, receives a definition of a logical router (e.g., through anapplication programming interface (API) and defines several routingcomponents for the logical router. Each of these routing components isseparately assigned a set of routes and a set of logical interfaces.

In some embodiments, the several routing components defined for alogical router includes one distributed routing component and severalcentralized routing components. In addition, the management plane ofsome embodiments defines a logical switch for handling communicationsbetween the components internal to the logical router (referred to as atransit logical switch). The distributed routing component and thetransit logical switch are implemented in a distributed manner bynumerous machines within the datacenter, while the centralized routingcomponents are each implemented on a single machine. Some embodimentsimplement the distributed components in the datapath of managedforwarding elements on the various machines, while the centralizedrouting components are implemented in VMs (or other data compute nodes)on their single machines. Other embodiments also implement thecentralized components in the datapath of their assigned machine.

The centralized components, in some embodiments, may be configured inactive-active or active-standby modes. In active-active mode, all of thecentralized components are fully functional at the same time, andtraffic can ingress or egress from the logical network through thecentralized components using equal-cost multi-path (ECMP) forwardingprinciples (balancing the traffic across the various centralizedcomponents). In this mode, each of the separate centralized componentshas its own network layer (e.g., IP) address and data link layer (e.g.,MAC) address for communicating with an external network. In addition,each of the separate centralized components has its own network layerand data link layer address for connecting to the transit logical switchin order to send packets to and receive packets from the distributedrouting component.

In some embodiments, the logical router is part of a two-tier logicalnetwork structure. The two-tier structure of some embodiments includes asingle logical router for connecting the logical network to a networkexternal to the datacenter (referred to as a provider logical router(PLR) and administrated by, e.g., the owner of the datacenter), andmultiple logical routers that connect to the single logical router anddo not separately communicate with the external network (referred to asa tenant logical router (TLR) and administrated by, e.g., differenttenants of the datacenter). Some embodiments implement the PLR inactive-active mode whenever possible, and only use active-standby modewhen stateful services (e.g., NAT, firewall, load balancer, etc.) areconfigured for the logical router.

For the PLR, some embodiments enable route exchange with the externalnetwork. Each of the centralized components of the PLR runs a dynamicrouting protocol process to advertise prefixes of the logical networkand receive routes towards the external network. Through a networkcontrol system of network controllers located both centrally in thedatacenter and on the machines that implement the logical network, theseroutes are propagated to the other centralized components and thedistributed routing component. Some embodiments use differentadministrative metrics in the routing information base (RIB) of thecentralized component for routes learned directly from the externalnetwork and routes learned from a different peer centralized componentthat learned the routes from the external network. Thus, a centralizedcomponent will prefer routes that it learned directly to routes thatinvolve redirection through peer centralized components of the logicalrouter. However, when the different centralized components haveinterfaces that are configured with different L3 connectivity towardsthe external network, some embodiments create dummy interfaces on thecentralized components that are used to redirect packets processed by afirst centralized component through a second centralized component tothe external network.

In active-standby mode, on the other hand, only one of the centralizedcomponents is fully operational at a time (the active component), andonly this component sends out messages to attract traffic. In someembodiments, the two components use the same network layer address (butdifferent data link layer addresses) for communicating with thedistributed component, and only the active component replies to addressresolution protocol (ARP) requests from this distributed component.Furthermore, only the active centralized component advertises routes tothe external network to attract traffic.

When the logical router is a TLR, some embodiments either use nocentralized components or two centralized components in active-standbymode when stateful services are configured for the logical router. TheTLR operates internally in the same manner as the PLR in active-standbymode, with each of the two centralized components having the samenetwork layer address, and only the active component responding to ARPrequests. To connect to the PLR, some embodiments also assign each ofthe two components a same network layer address (though different fromthe address used to connect to its own distributed component. Inaddition, the management plane defines a transit logical switch betweenthe distributed component of the PLR and the centralized components ofthe TLR.

In some cases, whether in active-active or active-standby mode, one (ormore) of the centralized router components will fail. This failure mayoccur due to the machine on which the component operates crashingcompletely, the data compute node or datapath software that implementsthe machine corrupting, the ability of the component to connect toeither the external network or through tunnels to other components ofthe logical network failing, etc. When the failed component is a standbyin active-standby mode, no action need be taken in some embodiments.Otherwise, when one of the centralized components fails, one of its peercomponents becomes responsible for taking over its communications.

In active-standby mode, the standby centralized router component isresponsible for taking over for the failed active centralized routercomponent. To do so, if the logical router is a PLR, the new activecomponent begins advertising routes to the external network so as toattract traffic from the external network (the failed component, if itsconnectivity to the external network remains, is responsible forstopping its own route advertisement so as to avoid attracting thistraffic). In addition, the new active component sends messages (e.g.,gratuitous ARP (GARP) replies) to the distributed routing component ofthe PLR that it is now responsible for the network layer address sharedbetween the two components. If the logical router is a TLR, this sameset of GARP replies are sent. In addition, to attract traffic from thePLR to which it connects, the new active component sends GARP replies tothe transit logical switch that connects it to the PLR.

For the active-active mode of some embodiments, the management planedesignates all of the centralized components for a logical router with aranking at the time they are created. This ranking is then used todetermine which of the peer components will take over for a failedcomponent. Specifically, in some embodiments the centralized componentwith the next-highest ranking to that of the failed component takes overfor the failed component. To take over, the overtaking componentidentifies the network layer address of the failed component thatcommunicates with the distributed component for the logical router, andsends GARP replies associating its own data link layer address with thenetwork layer address of the failed component.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a configuration view of a logical router, whichrepresents a logical network as designed by a user.

FIG. 2 illustrates a management plane view of the logical network ofFIG. 1 when the logical router is implemented in a centralized manner.

FIG. 3 illustrates a physical centralized implementation of the logicalrouter of FIG. 1 .

FIG. 4 illustrates a management plane view of the logical network ofFIG. 1 when the logical router is implemented in a distributed manner.

FIG. 5 illustrates a physical distributed implementation of the logicalrouter of FIG. 1 .

FIG. 6 conceptually illustrates a logical network with two tiers oflogical routers.

FIG. 7 illustrates the management plane view for the logical topology ofFIG. 6 when a TLR in the logical network is completely distributed.

FIG. 8 illustrates the management plane view for the logical topology ofFIG. 6 when the TLR in the logical network has a centralized component.

FIG. 9 conceptually illustrates a more detailed configuration of alogical network topology, including the network addresses and interfacesassigned by an administrator.

FIG. 10 illustrates the configuration of the logical topology of FIG. 9by the management plane.

FIG. 11 conceptually illustrates a process of some embodiments forconfiguring a PLR based on a user specification.

FIG. 12 conceptually illustrates a process of some embodiments forconfiguring a TLR based on a user specification.

FIG. 13 conceptually illustrates a physical implementation of themanagement plane constructs for the two-tiered logical network shown inFIG. 8 , in which the TLR and the PLR both include SRs as well as a DR.

FIGS. 14A-B illustrate examples of traffic that egresses from thelogical network (northbound traffic) and ingresses to the logicalnetwork (southbound traffic), respectively, for a logical topology witha single tier of logical routers.

FIGS. 15A-B illustrate examples of northbound and southbound traffic fora two-tier logical topology, with no centralized services provided inthe lower (TLR) tier.

FIGS. 16A-B illustrate examples of northbound and southbound traffic fora two-tier logical topology with centralized services provided in thelower (TLR) tier by SRs.

FIG. 17 conceptually illustrates the various stages of SR processing ofsome embodiments.

FIGS. 18 and 19 illustrate a single-tier logical network topology andthe management plane view of that topology that meets the requirementsfor the use of ECMP.

FIG. 20 illustrates a management plane view of the logical networktopology of FIG. 18 when the logical router is configured inactive-standby mode, rather than active-active (ECMP) mode.

FIG. 21 illustrates an example physical implementation of three gatewaymachines that host the three reSRs for a particular PLR.

FIG. 22 conceptually illustrates the result of one of the VMs thatimplements one of the SRs of FIG. 21 crashing.

FIG. 23 conceptually illustrates the result of complete tunnel failureat an MFE on the gateway machine that hosts one of the SRs of FIG. 21 .

FIG. 24 conceptually illustrates a process performed by a SR in case offailover of a peer SR.

FIG. 25 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

Some embodiments provide a two-tier logical router topology forimplementation in, e.g., a datacenter. These tiers include a top tier ofa provider logical router (PLR) and a lower tier of tenant logicalrouters (TLR), in some embodiments. The two-tiered structure enablesboth the provider (e.g., datacenter owner) and the tenant (e.g.,datacenter customer, often one of many such customers) control overtheir own services and policies. In some embodiments, the PLR layer isthe logical layer that interfaces with external physical networks, andtherefore dynamic routing protocols (e.g., BGP) may be configured on thePLR to enable the exchange of routing information with physical routersoutside the datacenter. Some embodiments also allow the configuration ofbidirectional forwarding detection (BFD) or similar protocols formonitoring whether physical network routers are up. Some datacenters maynot have multiple tenants, in which case the need for separate PLR andTLRs is removed. In such cases, some embodiments use a single-tierlogical router topology, with the single tier having the functionalityof PLRs. The two-tier logical topology of some embodiments is describedin greater detail in U.S. patent application Ser. No. 14/222,557, filedMar. 21, 2014, which is incorporated herein by reference.

In some embodiments, both PLRs and TLRs have the ability to supportstateless services (e.g., access control lists (ACLs)) as well asstateful services (e.g., firewalls). In addition, logical switches (towhich data compute nodes such as VMs may couple) may connect to either aPLR or a TLR. Furthermore, both PLRs and TLRs can be implemented ineither a distributed manner (e.g., with the logical router processingperformed in first-hop MFEs that physically couple directly to the datacompute nodes) or a centralized manner (with the logical routerprocessing performed in gateways for both north-south and east-westtraffic). For centralized implementations, as well as for thecentralized gateways by which PLRs interact with the physical networkeven when implemented in a distributed manner, both tiers of logicalrouters may be scaled out by using multiple physical boxes in order toprovide additional throughput (e.g., using equal-cost multi-path (ECMP)techniques) as well as for failure protection.

In some embodiments, the logical routers may only use stateful servicesif implemented at least partially in a centralized (e.g., clustered)manner (to avoid the need for state-sharing between the logical routerimplementations). In different embodiments, these gateways (that providecentralized aspects of logical routers, as well as which form theconnection to the external network for distributed PLRs) may beimplemented as virtual machines (sometimes referred to as Edge VMs), inother types of data compute nodes (e.g., namespaces), or by using theLinux-based datapath development kit (DPDK) packet processing software(e.g., as a VRF in the DPDK-based datapath).

The following introduces some of the terminology and abbreviations usedin the specification:

-   -   VNI (Virtual/Logical Network Identifier)—a unique identifier        (e.g., a 24-bit identifier) for a logical domain (e.g., a        logical switch)    -   PLR (Provider Logical Router—introduced above, a logical router        over which a service provider (e.g., datacenter operator) has        full control; interfaces directly with an external physical        network.    -   TLR (Tenant Logical Router)—a logical router over which a tenant        (e.g., a datacenter customer, a group within an enterprise,        etc.) has full control; connects to a PLR to access an external        physical network.    -   Distributed Logical Router—a logical router that supports        first-hop routing; that is, the logical router is implemented in        the managed forwarding elements to which the data compute nodes        directly couple.    -   Centralized Logical Router—a logical router that does not        support first hop-routing    -   Service Router (SR)—part of the realization of a logical router        that is used to provide centralized services; in some        embodiments, the SR is not exposed to the network manager APIs        except for troubleshooting purposes.    -   Distributed Router (DR)—part of the realization of a logical        router used to provide first-hop routing; in some embodiments,        the DR is also not exposed to the network manager APIs except        for troubleshooting purposes.    -   Uplink—refers to both (i) the northbound interface of a logical        router (directed towards the external physical network) and (ii)        a team of pNICs of a gateway.    -   Logical switch—a logical L2 broadcast domain.    -   Transit logical switch—a logical switch created automatically by        the network manager to connect SRs/DR of a TLR with the DR of a        DR; in some embodiments, a transit logical switch has no data        compute nodes (e.g., customer workload VMs) connected to it;        furthermore, in some embodiments, the transit logical switch is        not exposed to the network manager APIs except for        troubleshooting purposes    -   Context—a datapath representation of a logical router; in some        embodiments, the context may be a VRF, a namespace, or a VM    -   Transport Node, or Gateway—a node that terminates tunnels        defined by the network manager; in various embodiments, may be a        hypervisor-implemented virtual switch or a DPDK-based Edge Node;        in some embodiments, transport node may be used interchangeably        with datapath.    -   Deployment Container (DC), or Edge Cluster—a collection of        homogeneous nodes, the uplinks of which share the same L2        connectivity; in some embodiments, all nodes in a DC are of the        same type and belong to the same failure domain.    -   Edge Node—a node in a DC; may be a DPDK-based Edge or a        hypervisor-implemented virtual switch

The above introduces the concept of a two-tiered logical routerconfiguration as well as certain aspects of the logical routerconfiguration and implementation of some embodiments. In the following,Section I focuses on the overall high-level design of the logical routerof some embodiments, while Section II describes the configuration of thevarious logical router components. Section III then describes the packetprocessing through the various pipelines of some embodiments. Next,Section IV describes ECMP processing in the active-active configuration,while Section V describes the active-standby configuration. Section VIthen describes failover scenarios for the SRs. Finally, Section VIIdescribes the electronic system with which some embodiments of theinvention are implemented.

I. Logical Router and Physical Implementation

The following discussion describes the design of logical routers forsome embodiments as well as the implementation of such logical routersby the network controllers of some embodiments. As mentioned above, thelogical routers of some embodiments are designed such that they can beimplemented in either a distributed or centralized manner, they canscale out with or without stateful (or stateless) services, and so thatsuch services may be provided by either a VRF context in a datapath orby a virtual machine context.

Logical routers, in some embodiments, exist in three different forms.The first of these forms is the API view, or configuration view, whichis how the logical router is defined by a user, such as a datacenterprovider or tenant (i.e., a received definition of the logical router).The second view is the control plane, or management plane, view, whichis how the network controller internally defines the logical router.Finally, the third view is the physical realization, or implementationof the logical router, which is how the logical router is actuallyimplemented in the datacenter.

In the control plane view, the logical router of some embodiments mayinclude one or both of a single DR and one or more SRs. The DR, in someembodiments, spans managed forwarding elements (MFEs) that coupledirectly to VMs or other data compute nodes that are logicallyconnected, directly or indirectly, to the logical router. The DR of someembodiments also spans the gateways to which the logical router isbound. The DR, as mentioned above, is responsible for first-hopdistributed routing between logical switches and/or other logicalrouters that are logically connected to the logical router. The SRs ofsome embodiments are responsible for delivering services that are notimplemented in a distributed fashion (e.g., some stateful services).

A. Centralized Logical Router

FIGS. 1-3 illustrate the three different views of a centralized logicalrouter implementation. FIG. 1 specifically illustrates the configurationview, which represents a logical network 100 as designed by a user. Asshown, the logical router 115 is part of a logical network 100 thatincludes the logical router 115 and two logical switches 105 and 110.The two logical switches 105 and 110 each have VMs that connect tological ports. While shown as VMs in these figures, it should beunderstood that other types of data compute nodes (e.g., namespaces,etc.) may connect to logical switches in some embodiments. The logicalrouter 115 also includes two ports that connect to the external physicalnetwork 120.

FIG. 2 illustrates the management plane view 200 of the logical network100. The logical switches 105 and 110 are the same in this view as theconfiguration view, but the network controller has created two servicerouters 205 and 210 for the logical router 115. In some embodiments,these SRs operate in active-standby mode, with one of the SRs active andthe other operating as a standby (in case of the failure of the activeSR). Each of the logical switches 105 and 110 has a connection to eachof the SRs 205 and 210. If the logical network 100 included threelogical switches, then these three logical switches would each connectto both of the SRs 205 and 210.

Finally, FIG. 3 illustrates the physical centralized implementation ofthe logical router 100. As shown, each of the VMs that couples to one ofthe logical switches 105 and 110 in the logical network 100 operates ona host machine 305. The MFEs 310 that operate on these host machines arevirtual switches (e.g., OVS, ESX) that operate within the hypervisors orother virtualization software on the host machines. These MFEs performfirst-hop switching for the logical switches 105 and 110 for packetssent by the VMs of the logical network 100. The MFEs 310 (or a subset ofthem) also may implement logical switches (and distributed logicalrouters) for other logical networks if the other logical networks haveVMs that reside on the host machines 305 as well.

The two service routers 205 and 210 each operate on a different gatewaymachine 315 and 320. The gateway machines 315 and 320 are host machinessimilar to the machines 305 in some embodiments, but host servicerouters rather than user VMs. In some embodiments, the gateway machines315 and 320 each include an MFE as well as the service router, in orderfor the MFE to handle any logical switching necessary. For instance,packets sent from the external network 120 may be routed by the servicerouter implementation on the gateway and then subsequently switched bythe MFE on the same gateway.

The SRs may be implemented in a namespace, a virtual machine, or as aVRF in different embodiments. The SRs may operate in an active-active oractive-standby mode in some embodiments, depending on whether anystateful services (e.g., firewalls) are configured on the logicalrouter. When stateful services are configured, some embodiments requireonly a single active SR. In some embodiments, the active and standbyservice routers are provided with the same configuration, but the MFEs310 are configured to send packets via a tunnel to the active SR (or tothe MFE on the gateway machine with the active SR). Only if the tunnelis down will the MFE send packets to the standby gateway.

B. Distributed Logical Router

While the above section introduces a centralized implementation for alogical router, some embodiments use distributed logical routerimplementations that enable first-hop routing, rather than concentratingall of the routing functionality at the gateways. In some embodiments,the physical realization of a distributed logical router always has a DR(i.e., the first-hop routing). A distributed logical router will haveSRs if either (i) the logical router is a PLR, and therefore connects toexternal physical networks or (ii) the logical router has servicesconfigured that do not have a distributed implementation (e.g., NAT,load balancing, DHCP in some embodiments). Even if there are no statefulservices configured on a PLR, some embodiments use SRs in theimplementation to help with failure handling in the case of ECMP.

FIGS. 4 and 5 illustrate, respectively, the management plane view andphysical implementation for a distributed logical router. Theconfiguration view entered by the user is the same as that shown in FIG.1 for a centralized router, with the difference being that the user(e.g., administrator) denotes that the logical router will bedistributed. The control plane view 400 for the distributedimplementation illustrates that, in addition to the two service routers405 and 410, the control plane creates a distributed router 415 and atransit logical switch 420. The configuration of the northbound andsouthbound interfaces of the various router constructs 405-415 and theirconnections with the transit logical switch 420 will be described infurther detail below. In some embodiments, the management planegenerates separate routing information bases (RIBs) for each of therouter constructs 405-415. That is, in addition to having separateobjects created in the management/control plane, each of the routerconstructs 405 is treated as a separate router with separate routes. Thetransit logical switch 420 then has logical ports for each of theserouters, and each of the router constructs has an interface to thetransit logical switch.

FIG. 5 illustrates the physical distributed implementation of thelogical router 100. As in the centralized implementation, each of theVMs that couples to one of the logical switches 105 and 110 in thelogical network 100 operates on a host machine 505. The MFEs 510 performfirst-hop switching and routing for the logical switches 105 and 110 andfor the logical router 115 (in addition to performing switching and/orrouting for other logical networks). As shown in FIG. 5 , thedistributed router 415 is implemented across the MFEs 510 as well asgateways 515 and 520. That is, the datapaths (e.g., in the MFEs 510, ina similar MFE in the gateways 515 and 520 or in a different form factoron the gateways) all include the necessary processing pipelines for theDR 415 (and the transit logical switch 420). The packet processing ofsome embodiments will be described in greater detail below.

C. Multi-Tier Topology

The previous examples illustrate only a single tier of logical router.For logical networks with multiple tiers of logical routers, someembodiments may include both DRs and SRs at each level, or DRs and SRsat the upper level (the PLR tier) with only DRs at the lower level (theTLR tier). FIG. 6 conceptually illustrates a multi-tier logical network600 of some embodiments, with FIGS. 7 and 8 illustrating two differentmanagement plane views of the logical networks.

FIG. 6 conceptually illustrates a logical network 600 with two tiers oflogical routers. As shown, the logical network 600 includes, at thelayer 3 level, a provider logical router 605, several tenant logicalrouters 610-620. The first tenant logical router 610 has two logicalswitches 625 and 630 attached, with one or more data compute nodescoupling to each of the logical switches. For simplicity, only thelogical switches attached to the first TLR 610 are shown, although theother TLRs 615-620 would typically have logical switches attached (towhich data compute nodes couple).

In some embodiments, any number of TLRs may be attached to a PLR such asthe PLR 605. Some datacenters may have only a single PLR to which allTLRs implemented in the datacenter attach, whereas other datacenters mayhave numerous PLRs. For instance, a large datacenter may want to usedifferent PLR policies for different tenants, or may have too manydifferent tenants to attach all of the TLRs to a single PLR. Part of therouting table for a PLR includes routes for all of the logical switchdomains of its TLRs, so attaching numerous TLRs to a PLR creates severalroutes for each TLR just based on the subnets attached to the TLR. ThePLR 605, as shown in the figure, provides a connection to the externalphysical network 635; some embodiments only allow the PLR to providesuch a connection, so that the datacenter provider can manage thisconnection. Each of the separate TLRs 610-620, though part of thelogical network 600, are configured independently (although a singletenant could have multiple TLRs if they so chose).

FIGS. 7 and 8 illustrate different possible management plane views ofthe logical network 600, depending on whether or not the TLR 605includes a centralized component. In these examples, the routing aspectsof the TLR 605 are always distributed using a DR. However, if theconfiguration of the TLR 605 includes the provision of statefulservices, then the management plane view of the TLR (and thus thephysical implementation) will include active and standby SRs for thesestateful services.

Thus, FIG. 7 illustrates the management plane view 700 for the logicaltopology 600 when the TLR 605 is completely distributed. For simplicity,only details of the first TLR 610 are shown; the other TLRs will eachhave their own DR, as well as SRs in some cases. As in FIG. 4 , the PLR605 includes a DR 705 and three SRs 710-720, connected together by atransit logical switch 725. In addition to the transit logical switch725 within the PLR 605 implementation, the management plane also definesseparate transit logical switches 730-740 between each of the TLRs andthe DR 705 of the PLR. In the case in which the TLR 610 is completelydistributed (FIG. 7 ), the transit logical switch 730 connects to a DR745 that implements the configuration of the TLR 610. Thus, as will bedescribed in greater detail below, a packet sent to a destination in theexternal network by a data compute node attached to the logical switch625 will be processed through the pipelines of the logical switch 625,the DR 745 of TLR 610, the transit logical switch 730, the DR 705 of thePLR 605, the transit logical switch 725, and one of the SRs 710-720. Insome embodiments, the existence and definition of the transit logicalswitches 725 and 730-740 are hidden from the user that configures thenetwork through the API (e.g., an administrator), with the possibleexception of troubleshooting purposes.

FIG. 8 illustrates the management plane view 800 for the logicaltopology 600 when the TLR 605 has a centralized component (e.g., becausestateful services that cannot be distributed are defined for the TLR).In some embodiments, stateful services such as firewalls, NAT, loadbalancing, etc. are only provided in a centralized manner. Otherembodiments allow for some or all of such services to be distributed,however. As with the previous figure, only details of the first TLR 610are shown for simplicity; the other TLRs may have the same definedcomponents (DR, transit LS, and two SRs) or have only a DR as in theexample of FIG. 7 ). The PLR 605 is implemented in the same manner as inthe previous figure, with the DR 705 and the three SRs 710, connected toeach other by the transit logical switch 725. In addition, as in theprevious example, the management plane places the transit logicalswitches 730-740 between the PLR and each of the TLRs.

The partially centralized implementation of the TLR 610 includes a DR805 to which the logical switches 625 and 630 attach, as well as two SRs810 and 815. As in the PLR implementation, the DR and the two SRs eachhave interfaces to a transit logical switch 820. This transit logicalswitch serves the same purposes as the switch 725, in some embodiments.For TLRs, some embodiments implement the SRs in active-standby manner,with one of the SRs designated as active and the other designated asstandby. Thus, so long as the active SR is operational, packets sent bya data compute node attached to one of the logical switches 625 and 630will be sent to the active SR rather than the standby SR.

The above figures illustrate the management plane view of logicalrouters of some embodiments. In some embodiments, an administrator orother user provides the logical topology (as well as other configurationinformation) through an API. This data is provided to a managementplane, which defines the implementation of the logical network topology(e.g., by defining the DRs, SRs, transit logical switches, etc.). Inaddition, in some embodiments a user associates each logical router(e.g., each PLR or TLR) with a set of physical machines (e.g., apre-defined group of machines in the datacenter) for deployment. Forpurely distributed routers, such as the TLR 605 as implemented in FIG. 7, the set of physical machines is not important, as the DR isimplemented across the managed forwarding elements that reside on hostsalong with the data compute nodes that connect to the logical network.However, if the logical router implementation includes SRs, then theseSRs will each be deployed on specific physical machines. In someembodiments, the group of physical machines is a set of machinesdesignated for the purpose of hosting SRs (as opposed to user VMs orother data compute nodes that attach to logical switches). In otherembodiments, the SRs are deployed on machines alongside the user datacompute nodes.

In some embodiments, the user definition of a logical router includes aparticular number of uplinks. Described herein, an uplink is anorthbound interface of a logical router in the logical topology. For aTLR, its uplinks connect to a PLR (all of the uplinks connect to thesame PLR, generally). For a PLR, its uplinks connect to externalrouters. Some embodiments require all of the uplinks of a PLR to havethe same external router connectivity, while other embodiments allow theuplinks to connect to different sets of external routers. Once the userselects a group of machines for the logical router, if SRs are requiredfor the logical router, the management plane assigns each of the uplinksof the logical router to a physical machine in the selected group ofmachines. The management plane then creates an SR on each of themachines to which an uplink is assigned. Some embodiments allow multipleuplinks to be assigned to the same machine, in which case the SR on themachine has multiple northbound interfaces.

As mentioned above, in some embodiments the SR may be implemented as avirtual machine or other container, or as a VRF context (e.g., in thecase of DPDK-based SR implementations). In some embodiments, the choicefor the implementation of an SR may be based on the services chosen forthe logical router and which type of SR best provides those services.

In addition, the management plane of some embodiments creates thetransit logical switches. For each transit logical switch, themanagement plane assigns a unique VNI to the logical switch, creates aport on each SR and DR that connects to the transit logical switch, andallocates an IP address for any SRs and the DR that connect to thelogical switch. Some embodiments require that the subnet assigned toeach transit logical switch is unique within a logical L3 networktopology having numerous TLRs (e.g., the network topology 600), each ofwhich may have its own transit logical switch. That is, in FIG. 8 ,transit logical switch 725 within the PLR implementation, transitlogical switches 730-740 between the PLR and the TLRs, and transitlogical switch 820 (as well as the transit logical switch within theimplementation of any of the other TLRs) each require a unique subnet.Furthermore, in some embodiments, the SR may need to initiate aconnection to a VM in logical space, e.g. HA proxy. To ensure thatreturn traffic works, some embodiments avoid using link local IPaddresses.

Some embodiments place various restrictions on the connection of logicalrouters in a multi-tier configuration. For instance, while someembodiments allow any number of tiers of logical routers (e.g., a PLRtier that connects to the external network, along with numerous tiers ofTLRs), other embodiments only allow a two-tier topology (one tier ofTLRs that connect to the PLR). In addition, some embodiments allow eachTLR to connect to only one PLR, and each logical switch created by auser (i.e., not a transit logical switch) is only allowed to connect toone PLR or one TLR. Some embodiments also add the restriction thatsouthbound ports of a logical router must each be in different subnets.Thus, two logical switches may not have the same subnet if connecting tothe same logical router. Lastly, some embodiments require that differentuplinks of a PLR must be present on different gateway machines. Itshould be understood that some embodiments include none of theserequirements, or may include various different combinations of therequirements.

II. SR and DR Configuration

When a user configures a logical router, this configuration is used bythe management plane to configure the SRs and DR for the logical router.For instance, the logical router 115 of FIG. 1 has four interfaces (twoto the logical switches, and two uplinks). However, its distributedmanagement plane implementation in FIG. 4 includes a DR with threeinterfaces and SRs with two interfaces each (a total of seveninterfaces). The IP and MAC addresses and other configuration detailsassigned to the four interfaces as part of the logical routerconfiguration are used to generate the configuration for the variouscomponents of the logical router.

In addition, as part of the configuration, some embodiments generate arouting information base (RIB) for each of the logical routercomponents. That is, although the administrator defines only a singlelogical router, the management plane and/or control plane of someembodiments generates separate RIBs for the DR and for each of the SRs.For the SRs of a PLR, in some embodiments the management plane generatesthe RIB initially, but the physical implementation of the SR also runs adynamic routing protocol process (e.g., BGP, OSPF, etc.) to supplementthe RIB locally.

Some embodiments include several types of routes in the RIB of a logicalrouters, and therefore in the RIBs of its component routers. All routes,in some embodiments, include administrative distance values, used todetermine priority, with larger values indicating lower priority typesof route (i.e., if two routes exist for the same prefix, the one with alower distance value is used). If multiple routes for the same prefixare in the RIB with the same distance value, traffic to these prefixesis spread across the different routes (e.g., using ECMP principles tobalance the traffic evenly).

-   -   connected (0): prefixes configured on the logical router's ports    -   static (1): configured by the administrator/user    -   management plane internal (10): default routes—when a TLR is        connected to a PLR, a default route pointing to the PLR is added        to the RIB of the TLR; when a logical switch is connected to a        TLR, the user allows the subnet to be redistributed, and the        subnet is not NAT'ed, a default route pointing to the TLR for        the subnet is added to the RIB of the PLR    -   EBGP (20): the next four types are routes learned through        dynamic routing protocols    -   OSPF internal (30)    -   OSPF external (110)    -   IBGP (200).

It should be understood that not all logical routers will include bothBGP and OSPF routes in some embodiments, and some logical routers mayinclude neither. For instance, a logical router that does not include aconnection to external networks may not use any routing protocol, andsome logical routers may run only one type of route-sharing protocol,rather than both BGP and OSPF.

In addition, in some embodiments, the SRs of the PLRs (that use thedynamic routing protocols) merge the RIB received from the centralizedcontrollers (containing static, connected, and management plane internalroutes) with the routes learned from the physical routers (via thedynamic routing protocols). The SR locally calculates its FIB based onthe incorporation of these dynamic routes in order to expedite routeconvergence, rather than sending the learned routes back to thecentralized controller for recalculation. For the DRs, the centralizedcontrollers of some embodiments pushes down the entire RIB, with a localcontrol plane calculating the FIB.

A. DR Configuration

In some embodiments, the DR is always located on the southbound side(i.e., facing the data compute nodes of the logical network, rather thanfacing the external physical network) of the logical routerimplementation. Unless the logical router has no centralized component,the uplinks of the logical router will not be configured for the DR,whose northbound interfaces instead couple to the transit logical switchthat is part of the logical router.

FIG. 9 conceptually illustrates the more detailed configuration of alogical network topology 900, including the network addresses andinterfaces assigned by an administrator. As shown, the logical switches905 and 910 are each assigned their own subnets, 1.1.1.0/24 and1.1.2.0/24, and all of the data compute nodes attached to the logicalswitches 905 will have IP addresses in the corresponding subnet. Thelogical router 915 has an interface L1 to the first logical switch 905,with an IP address of 1.1.1.253 that is the default gateway for the datacompute nodes in the subnet 1.1.1.0/24. The logical router 915 also hasa second interface L2 to the second logical switch 910, with an IPaddress of 1.1.2.253 that is the default gateway for the data computenodes in the subnet 1.1.2.0/24.

The northbound side of the logical router 915 has two uplinks, U1 andU2. The first uplink U1 has an IP address of 192.168.1.252 and connectsto a first physical router 920 with an IP address of 192.168.1.252. Thesecond uplink U2 has an IP address of 192.168.2.253 and connects to asecond physical router 925 with an IP address of 192.168.2.252. Thephysical routers 920 and 925 are not actually part of the logicalnetwork, but rather connect the logical network to the external network.Though in the illustrated case each of the uplinks connects to a single,different physical router, in some cases each of the uplinks willconnect to the same set of several physical routers. That is, both U1and U2 might both connect to both of the physical routers 920 and 925.Some embodiments require that each of the external routers to which theuplinks connect provide the same connectivity, although this is not thecase in the illustrated example. Instead, the first physical router 920connects to the subnet 10.0.0.0/8, while the second router 925 connectsto both the subnet 10.0.0.0/8 and 11.0.0.0/8.

For a logical router with a distributed component, some embodimentsconfigure the DR as follows. The southbound interfaces are configured inthe same way as the southbound interfaces of the logical router. Theseinterfaces are those that connect to a logical switch in the logicaltopology, or to a lower-level logical router (e.g., the southboundinterfaces of a PLR may connect to TLRs). The DR of some embodiments isallocated a single northbound interface, which is assigned an IP addressand a MAC address. Assuming the logical router has one or more SRs, thenorthbound interface of the DR connects to a transit logical switch.

The RIB of the DR is assigned connected routes based on the subnetsconfigured on its various southbound and northbound interfaces. Theseare the subnets configured for (i) the transit logical switch configuredbetween the DR and SR components of the logical router, and (ii) anylogical switches on its southbound interfaces. These logical switches onthe southbound interfaces may be user-defined logical domains to whichdata compute nodes connect, or transit logical switches located betweenthe DR of a PLR and any TLRs that connect to the PLR.

In addition, any static routes that egress from an uplink of the logicalrouter are included in the RIB of the DR; however, these routes aremodified such that the next-hop IP address is set to that of theuplink's SR. For example, a static route “a.b.c.0/24 via 192.168.1.252”(192.168.1.252 being an address of an external physical network router)is modified to be “a.b.c.0/24 via [IP of SR southbound interface]”.Static routes that egress from a southbound interface of the logicalrouter, on the other hand, are included in the RIB of the DR unmodified.In some embodiments, for each SR of the logical router, a default routeof the type management plane internal is added to the RIB of the DR.Instead, in other embodiments, dynamic routes learned by a particular SRare added to the RIB, with the next-hop IP address modified to be the IPof the southbound interface of the particular SR. This is an alternativeto the default route, because the management plane internal type wouldotherwise have a higher priority than the dynamic routes learned by theSR. However, for TLRs, the SRs do not run a dynamic routing protocol insome embodiments, so the default route with a next-hop IP addresspointing to the interface of the active SR is used instead.

FIG. 10 illustrates the configuration 1000 of the logical topology 900by the management plane. As shown, the logical switches 905 and 910 areconfigured as indicated by the user configuration. As in the previousexamples, the logical router 915 includes a DR 1005, two SRs 1010 and1015, and a transit logical switch 1020. The DR is assigned the twosouthbound interfaces of the logical router 905, which connect to thelogical switches 905 and 910. The transit logical switch is assigned asubnet of 192.168.100.0/24, which needs to satisfy the requirement thatit be unique among the logical switches that logically connect (directlyor indirectly) to the logical router 905. Each of the three managementplane router constructs 1005-1015 also includes an interface thatconnects to the transit logical switch, and has an IP address in thesubnet of the transit logical switch. The northbound interfaces U1 andU2 are assigned to the two SRs 1010 and 1015, the configuration of whichis described below.

Using the rules of some embodiments described above for generating theRIB, the RIB of the DR 1005 includes the following routes:

-   -   1.1.1.0/24 output to L1    -   1.1.2.0/24 output to L2    -   192.168.100.0/24 output to DRP1    -   192.168.1.0/24 via IP1    -   192.168.2.0/24 via IP2    -   10.0.0.0/8 via IP1    -   10.0.0.0/8 via IP2    -   11.0.0.0/8 via IP2    -   0.0.0.0/0 via IP1    -   0.0.0.0/0 via IP2

The above routes include three connected routes, for the logical switchdomains connected to the DR (1.1.1.0/24, 1.1.2.0/24, and192.168.100.0/24). In addition, the subnet on which the first uplink islocated (192.168.1.0/24) is reached via the southbound interface of thefirst SR 1010 (IP1), while the subnet on which the second uplink islocated (192.168.2.0/24) is reached via the southbound interface of thesecond SR 1015 (IP2). In addition, three static routes have been addedby the user for the logical router 915, which the management planeautomatically modifies for the DR 1005. Specifically, the routes includethe network 10.0.0.0/8 via the southbound interface of either of theSRs, and the network 11.0.0.0/8 via the southbound interface of SR2.Lastly, default routes pointing to these same southbound interfaces areincluded. The IP addresses IP1, IP2, and IP3 that are created by themanagement plane for the ports of the logical router constructs thatinterface with the transit logical switch all are in the subnet192.168.100.0/24.

In addition to configuring the RIB of the DR, the management plane alsoassigns MAC addresses to the DR interfaces in some embodiments. In someembodiments, some or all of the physical routing elements (e.g.,software modules) in the physical network that implement the DRfunctionality only support a single MAC address. In this case, becausethe MAC of a DR port may come from that of a logical router port visibleto users, this imposes requirements on how the management planeallocates MAC addresses for the logical router ports. Thus, in someembodiments, all DR/SR ports that connect to any logical switch whichhas user data compute nodes or SRs connected must share a common MACaddress. In addition, if a DR/SR port is connected to another DR/SR orto a physical network, this port is assigned a unique MAC address insome embodiments (this assignment rule ignores the transit logicalswitch when determining whether a DR/SR port is connected to anotherDR/SR port)

B. SR Configuration

As with the DR of a logical router, the management plane also configureseach SR of the logical router with a separate RIB and interfaces. Asdescribed above, in some embodiments SRs of both PLRs and TLRs maydeliver services (i.e., functionalities beyond simply routing, such asNAT, firewall, load balancing, etc.) and the SRs for PLRs also providethe connection between the logical network and external physicalnetworks. In some embodiments, the implementation of the SRs is designedto meet several goals. First, the implementation ensures that theservices can scale out—that is, the services assigned to a logicalrouter may be delivered by any of the several SRs of the logical router.Second, some embodiments configure the SR in such a way that the servicepolicies may depend on routing decisions (e.g., interface-based NAT).Third, the SRs of a logical router have the ability to handle failure(e.g., of the physical machine on which an SR operates, of the tunnelsto that physical machine, etc.) among themselves without requiring theinvolvement of a centralized control plane or management plane (thoughsome embodiments allow the SRs to operate at reduced capacity or in asuboptimal manner). Finally, the SRs ideally avoid unnecessaryredirecting amongst themselves. That is, an SR should forward packets tothe external physical network if it has the ability do so locally, onlyforwarding the packet to a different SR if necessary. Of course, theforwarding between SRs should avoid packet loops.

As shown in FIG. 10 , each SR has one southbound interface that connectsto the transit logical switch 1020 that resides between the SRs and theDR. In addition, in some embodiments, each SR has the same number ofnorthbound interfaces as the logical router. That is, even though onlyone uplink may be assigned to the physical machine on which the SRoperates, all of the logical router interfaces are defined on the SR.However, some of these interfaces are local interfaces while some ofthem are referred to as dummy interfaces.

The local northbound interfaces, in some embodiments, are those throughwhich a packet can egress directly from the SR (e.g., directly to thephysical network). An interface configured based on the uplink (or oneof the uplinks) assigned to the SR is a local interface. On the otherhand, an interface configured based on one of the other uplinks of thelogical router assigned to a different SR is referred to as a dummyinterface. Providing the SR with configuration for the dummy interfacesallows for the first-hop MFEs to send packets for any of the uplinks toany of the SRs, with that SR able to process the packets even if thepacket is not destined for its local interface. Some embodiments, afterprocessing a packet at one of the SRs for a dummy interface, forward thepacket to the appropriate SR where that interface is local, in order forthe other SR to forward the packet out to the external physical network.The use of dummy interfaces also allows the centralized controller (orset of controllers) that manages the network to push service policiesthat depend on routing decisions to all of the SRs, thereby allowingservices to be delivered by any of the SRs.

As discussed below in Section IV, in some embodiments the SRs exchangerouting information with the physical network (e.g., using a routeadvertisement protocol such as BGP or OSPF). One goal of this routeexchange is that irrespective of which SR routes a packet towards thephysical network, the routing decision should always point to either alocal interface of the SR or a dummy interface that corresponds to anuplink of the logical router on a different SR. Thus, the policiesassociated with the logical router uplink can be applied by the SR evenwhen the uplink is not assigned to that SR, enabling the scale out ofstateful services. In some embodiments, the routes received from a peerSR will have a larger distance value than routes learned directly from aphysical next-hop router, thereby ensuring that a SR will send a packetto its peer SR only when it cannot send the packet directly to aphysical network router.

For a logical router that has one or more centralized components, someembodiments configure the SR as follows. For northbound interfaces, theSR has the same number of such interfaces as the logical router, andthese interfaces each inherit the IP and MAC address of thecorresponding logical router interfaces. A subset of these interfacesare marked as local interfaces (those for which the uplink is assignedto the SR), while the rest of the interfaces are marked as dummyinterfaces. In some embodiments, the service policies defined for thelogical router are pushed equivalently to all of the SRs, as these areconfigured in the same way from the network and interface perspective.The dynamic routing configuration for a particular logical routerport/uplink are transferred to the local interface of the SR to whichthat particular uplink is assigned.

Each SR, as mentioned, is assigned a single southbound interface (also alocal interface) that connects to a transit logical switch, with eachSR's southbound interface connecting to the same transit logical switch.The IP addresses for each of these southbound interfaces is in the samesubnet as the northbound interface assigned to the DR (that of thetransit logical switch). Some embodiments differentiate the assignmentof IP addresses between the SRs depending on whether the SRs are inactive-active or active-standby mode. For active-active mode (i.e., whenall of the SRs are treated as equals for routing purposes), different IPand MAC addresses are assigned to the southbound interfaces of all ofthe SRs. On the other hand, in active-standby mode, the same IP is usedfor both of the southbound interfaces of the two SRs, while each of theinterfaces is assigned a different MAC address.

As indicated in the above subsection regarding DRs, users may configurestatic routes for the logical router. A static route (or a connectedroute) of the logical router that egresses from an uplink is copied tothe RIB of the SR. The distance metric for such a route is unmodified ifthe uplink through which the route egresses is assigned to the SR;however, if the uplink is a dummy interface on the SR, then someembodiments add a value to this metric so that the SR will prefer aroute that egresses from its local interface when the network can bereached without redirecting the packet to a different SR through a dummyinterface. In addition, the SRs (of a top-level logical router) maylearn dynamic routes and place these in their RIB (though someembodiments perform this locally, without involving the centralizedcontrollers). In some embodiments, the dynamic routes learned from peerSRs are installed without this adjustment of the distance metric,because by default the metric for routes learned from IBGP (SR to SRpeering) or OSPF are larger than the metric for routes learned fromEBGP.

For each southbound interface of the logical router, some embodimentsadd a route for the corresponding network to the RIB of each SR. Thisroute points to the northbound DR interface as its next-hop IP address.Furthermore, any other routes configured for the logical router thategress from the southbound interface are copied to the SR with the samenorthbound DR interface as the next-hop IP address.

Returning to the example of FIG. 10 , as the logical router 915 has twouplinks, the management plane defines two service routers 1010 and 1015.The first service router 1010 has a local interface for U1 and a dummyinterface for U2, referred to as U2′. Similarly, the second servicerouter 1015 has a local interface for U2 and a dummy interface, U1′, forthe first uplink U1. Each of these SRs is assigned a southboundinterface, with different IP and MAC addresses (as the SRs are in anactive-active configuration). The IP addresses IP1 (for the first SR1010) and IP2 (for the second SR 1015) are in the subnet 192.1.100.0/24,as is IP3 (the northbound interface of the DR 1005).

Using the rules of some embodiments, and assuming the a routing protocol(e.g., BGP) is enabled for the SRs, the RIB of the first SR 1010 willinclude the following routes:

-   -   10.0.0.0/8 output to U1 via 192.168.1.252, metric 20 (via EBGP)    -   10.0.0.0/8 output to U2′ via 192.168.2.252, metric 200 (via        IBGP)    -   11.0.0.0/8 output to U2′ via 192.168.2.252, metric 200 (via        IBGP)    -   192.168.1.0/24 output to U1, metric 0 (connected)    -   192.168.100.0/24 output to SRP1, metric 0 (connected)    -   1.1.1.0/24 via IP3, metric 10 (management plane internal)    -   1.1.2.0/24 via IP3, metric 10 (management plane internal)

Similarly, the RIB of the second SR 1015 will include the followingroutes:

-   -   10.0.0.0/8 output to U2 via 192.168.2.252, metric 20 (via EBGP)    -   10.0.0.0/8 output to U1′ via 192.168.1.252, metric 200 (via        IBGP)    -   11.0.0.0/8 output to U2 via 192.168.2.252, metric 20 (via EBGP)    -   192.168.2.0/24 output to U2, metric 0 (connected)    -   192.168.100.0/24 output to SRP2, metric 0 (connected)    -   1.1.1.0/24 via IP3, metric 10 (management plane internal)    -   1.1.2.0/24 via IP3, metric 10 (management plane internal)

C. Management Plane Processes

FIG. 11 conceptually illustrates a process 1100 of some embodiments forconfiguring a PLR based on a user specification. In some embodiments,the process 1100 is performed by the management plane (e.g., a set ofmodules at a centralized controller that manages the networks of adatacenter). The management plane performs the configuration process,then uses a centralized control plane of the controller (or of adifferent network controller) to distribute the data to various localcontrol planes on the various host machines that implement theconfigured logical router.

As shown, the process 1100 begins by receiving (at 1105) a specificationof a PLR. The specification of a PLR (or definition of the PLR) is basedon administrator input to define the PLR (e.g., an administratoremployed by the owner of the datacenter). In some embodiments, thisspecification includes definitions of any services the PLR shouldprovide, whether the PLR will be configured in active-active oractive-standby mode (though some embodiments automatically useactive-active mode unless stateful services are configured), how manyuplinks are configured for the PLR, the IP and MAC addresses of theuplinks, the L2 and L3 connectivity of the uplinks, the subnets of anysouthbound interfaces of the PLR (one interface if the PLR is intendedfor a two-tier topology, and any number of interfaces if user logicalswitches will connect directly in a single-tier topology), any staticroutes for the RIB of the PLR, as well as other data. It should beunderstood that different embodiments may include different combinationsof the listed data or other data in the configuration data for a PLR.

The process 1100 then defines (at 1110) a DR using this configurationdata. This assumes that the PLR will not be completely centralized, inwhich case no DR is generated by the management plane. For thesouthbound interface of the DR, the management plane uses the southboundinterface configuration of the PLR. That is, the IP address and MACaddress for the DR are those specified for the logical router.

In addition, the process assigns (at 1115) each uplink specified for thePLR to a gateway machine. As described above, some embodiments allow (orrequire) the user to specify a particular set of physical gatewaymachines for the location of the SRs of the logical router. In someembodiments, the set of gateway machines might be together within aparticular rack or group of racks of servers, or are otherwise related,with tunnels connecting all of the machines in a set. The managementplane then assigns each of the uplinks to one of the gateway machines inthe selected set. Some embodiments allow multiple uplinks to be assignedto the same gateway machine (so long as the logical router does not haveonly two uplinks configured in active-standby mode), while otherembodiments only allow a single uplink per gateway machine for the PLRirrespective of whether in active-active or active-standby.

After assigning the uplinks to gateway machines, the process 1100defines (at 1120) a SR on each of the selected gateway machines. Foreach SR, the process uses the configuration for the uplink assigned tothat gateway machine as the configuration for the northbound interfaceof the SR. This configuration information includes the IP and MACaddress of the uplink, as well as any uplink-specific policies. Itshould be understood that, for situations in which different policiesand/or L3 connectivity are allowed and used between the differentuplinks, some embodiments also configure dummy interfaces on the SRs inorder to redirect packets if needed.

The process additionally defines (at 1125) a transit logical switch toconnect the defined SRs and DR. In some embodiments, the managementplane assigns a unique VNI (logical switch identifier) to the transitlogical switch. In addition, some embodiments require that the subnetassigned to the transit logical switch be unique among the logicalnetwork topology. As such, the transit logical switch must use a subnetdifferent from any user-defined logical switches that interface directlywith the PLR, as well as all transit logical switches between the PLRand any TLRs that connect to the PLR, all transit logical switcheswithin these TLRs, and any user-defined logical switches that connect tothese TLRs.

Next, the process 1100 assigns (at 1130) a northbound interface to theDR. The northbound interface, in some embodiments, is assigned both aMAC address and an IP address (used for packets sent internally betweenthe components of the PLR). In some embodiments, the IP address is inthe subnet that was assigned to the transit logical switch defined at1125. The configuration of the transit logical switch includes anassociation of this MAC address with one of its logical ports.

The process then determines (at 1135) whether the PLR is configured inactive-active mode (or active-standby mode). As noted above, in someembodiments, this determination is made by the administrator as part ofthe configuration settings for the PLR. In other embodiments, themanagement plane automatically defines the SRs in active-activeconfiguration for PLRs unless stateful services are set up, in whichcase the SRs are defined in active-standby mode.

When the PLR is configured in active-standby mode, the process assigns(at 1140) southbound interfaces of each of the two SRs (or more than twoSRs, if there are multiple standbys). In the active-standby case, thesesouthbound interfaces all have the same IP address, which is in thesubnet of the transit logical switch defined at operation 1125. Althoughthe two interfaces receive the same IP address, some embodiments assigndifferent MAC addresses, so as to differentiate the two as destinationsfor northbound packets routed by the DR. In other embodiments, the sameMAC addresses are used as well, with different mechanisms in the case offailover used as described below.

The process then assigns (at 1145) one of the SRs as active and one ofthe SRs as standby. Some embodiments make this determination randomly,while other embodiments attempt to balance the assignment of active andstandby SRs across the gateway machines, as described in greater detailin U.S. Patent Publication 2015/0063364, filed Jan. 28, 2014, which isincorporated herein by reference. The SR assigned as active will respondto ARP requests for the southbound interface, and will advertiseprefixes to the external physical network from its northbound interface.The standby SR, on the other hand, will not respond to ARP requests (soas to avoid receiving northbound traffic), and will not advertiseprefixes (but will maintain a BGP session in order to receive routesfrom the external network in case of failure of the active SR.

Lastly, the process 1100 generates (at 1150) separate RIBs for the DRand for each of the SRs. The separate RIBs are generated based on theconfiguration data in the manner described in the previous subsections,as well as below in Section V. The process then ends. In someembodiments, the management plane also calculates the FIB centrally,while in other embodiments the local control planes (operating on thehost and gateway machines) performs the RIB traversal to generate theFIB to use in actual forwarding of packets by the logical routercomponents. In either case, the RIB is updated on the SRs based on thedynamic routes learned from the external network, and that data ispropagated to the DR via central controllers. The calculation of the FIBby network controllers of some embodiments is described in greaterdetail in U.S. patent application Ser. No. 14/214,545, filed Mar. 14,2014, which is incorporated herein by reference.

On the other hand, when the PLR is configured in active-active (ECMP)mode, the process assigns (at 1155) southbound interfaces of each of theSRs. In the active-active cases, these southbound interfaces are eachassigned different IP addresses in the subnet of the transit logicalswitch defined at operation 1125, as well as different MAC addresses.With different IP addresses, each of the SRs can handle northboundpackets based on the IP address selected for a given packet by the DRpipeline in a host machine.

Next, the process assigns (at 1160) ranks to the SRs. As described indetail below, the SRs use the ranks in case of failover to determinewhich SR will take over responsibilities for a failed SR. In someembodiments, the next-highest ranked SR takes over for a failed SR bytaking over its southbound interfaces so as to attract northboundtraffic that would otherwise be sent to the IP address of the failed SR.

Finally, the process generates (at 1165) separate RIBs for the DR andfor each of the SRs. The separate RIBs are generated based on theconfiguration data in the manner described in the previous subsections,as well as below in Section IV. The process then ends. In someembodiments, the management plane also calculates the FIB centrally,while in other embodiments the local control planes (operating on thehost and gateway machines) performs the RIB traversal to generate theFIB to use in actual forwarding of packets by the logical routercomponents. In either case, the RIB is updated on the SRs based on thedynamic routes learned from the external network, and that data ispropagated to the DR via central controllers.

The above description of FIG. 11 indicates the operations of themanagement plane to generate the various components for a PLR (uppertier logical router). FIG. 12 conceptually illustrates a process 1200 ofsome embodiments for configuring a TLR based on a user specification. Insome embodiments, the process 1200 is performed by the management plane(e.g., a set of modules at a centralized controller that manages thenetworks of a datacenter). The management plane performs theconfiguration process, then uses a centralized control plane of thecontroller (or a different network controller) to distribute the data tovarious local control planes on the various host machines that implementthe configured logical router.

As shown, the process begins by receiving (at 1205) a specification of aTLR. The specification of a TLR (or definition of the TLR) is based onadministrator input to define the TLR (e.g., an administrator employedby a tenant of the datacenter). In some embodiments, this specificationincludes definitions of any services the TLR should provide, which PLRthe TLR should connect to through its uplink, any logical switches thatconnect to the TLR, IP and MAC addresses for the interfaces of the TLR,any static routes for the RIB of the TLR, as well as other data. Itshould be understood that different embodiments may include differentcombinations of the listed data or other data in the configuration datafor the TLR.

The process 1200 then determines (at 1210) whether the TLR has acentralized component. In some embodiments, if the TLR does not providestateful services, then no SRs are defined for the TLR, and it isimplemented only in a distributed manner. On the other hand, someembodiments require SRs in active-standby mode when stateful servicesare provided, as shown in this figure.

When the TLR does not provide stateful services or otherwise require acentralized component, the process defines (at 1215) a DR using thespecification of the logical router for both the southbound andnorthbound interfaces. The DR may have numerous southbound interfaces,depending on how many logical switches are defined to connect to theTLR. On the other hand, some embodiments restrict TLRs to a singlenorthbound interface that sends packets to and receives packets from aPLR. The process also generates (at 1220) a RIB for the DR. The RIB forthe DR will include all of the routes for the logical router, generatedas described above.

On the other hand, when the TLR provides stateful services or requires acentralized component for other reasons, the process defines (at 1225) aDR using the received configuration data. For the southbound interfacesof the DR, the management plane uses the southbound interfaceconfigurations of the TLR. That is, the IP address and MAC address foreach southbound interface are those specified for the ports of thelogical router to which the various logical switches couple.

In addition, the process assigns (at 1230) the uplink specified for theTLR to two gateway machines. While some embodiments allow TLRs tooperate in active-active mode with multiple uplinks, the process 1200 isfor embodiments that restrict the TLRs to a single uplink inactive-standby mode. As described above, some embodiments allow (orrequire) the user to specify a particular set of physical gatewaymachines for the location of the SRs of the logical router. In someembodiments, the set of gateway machines might be together within aparticular rack or group of racks of servers, or are otherwise related,with tunnels connecting all of the machines in a set. The managementplane then assigns the uplink to two of the gateway machines in theselected set.

After assigning the uplinks to gateway machines, the process 1200defines (at 1235) a SR on each of the two gateway machines. For each SR,the management plane uses the configuration for the single uplink as theconfiguration for the northbound interface of the SR. As there is onlyone northbound interface, the process applies the same configuration toboth of the SRs. That is, not only is the same IP address used for bothnorthbound interfaces, but the services on the interfaces are configuredin the same manner as well. However, different MAC addresses are usedfor the northbound interfaces, so as to differentiate the active andstandby SRs.

The process additionally defines (at 1240) a transit logical switch toconnect the defined SRs and DR. In some embodiments, the managementplane assigns a unique VNI (logical switch identifier) to the transitlogical switch. In addition, some embodiments require that the subnetassigned to the transit logical switch be unique among the logicalnetwork topology. As such, the management plane must assign the transitlogical switch a subnet different than any of the user-defined logicalswitches that interface with the TLR, as well as any transit logicalswitches between the TLR (or other TLRs) and the PLR, as well as alltransit logical switches within other TLRs that connect to the same PLR,the transit logical switch within the PLR, and the user-defined logicalswitches that connect to the other TLRs.

Next, the process assigns (at 1245) a northbound interface to the DR.This interface, in some embodiments, is assigned both a MAC address andan IP address (used for packets sent internally between the componentsof the TLR). In some embodiments, the IP address is in the same subnetthat was assigned to the transit logical switch at 1140. The processalso assigns (at 1250) southbound interfaces of each of the two SRs. Asthis is an active-standby configuration, these southbound interfaceshave the same IP address, which is in the subnet of the transit logicalswitch defined at operation 1140. Although the two interfaces receivethe same IP address, some embodiments assign different MAC addresses, soas to differentiate the two as destinations for northbound packetsrouted by the DR. In other embodiments, the same MAC addresses are usedas well, with different mechanisms in the case of failover used asdescribed below.

The process 1200 then assigns (at 1255) one of the SRs as active and oneof the SRs as standby. Some embodiments make this determinationrandomly, while other embodiments attempt to balance the assignment ofactive and standby SRs across the gateway machines. The SR assigned asactive will respond to ARP requests for the southbound (from the DR ofthis TLR) and northbound (from the DR of the PLR) interfaces. Thestandby SR, on the other hand, will not respond to ARP requests (so asto avoid receiving northbound or southbound traffic).

Next, the process generates (at 1260) separate RIBs for the DR and foreach of the SRs. The separate RIBs are generated based on theconfiguration data in the manner described in the previous subsections,as well as below in Section IV. In some embodiments, the managementplane also calculates the FIB centrally, while in other embodiments thelocal control planes (operating on the host and gateway machines)performs the RIB traversal to generate the FIB to use in actualforwarding of packets by the logical router components. In either case,the RIB is updated on the SRs based on the dynamic routes learned fromthe external network, and that data is propagated to the DR via centralcontrollers.

Irrespective of whether the TLR is generated with or without SRs, theprocess 1200 defines (at 1265) another transit logical between the TLRand the PLR to which it connects. This transit logical switch has aunique VNI, and a subnet to which the uplink IP address of the TLRbelongs. In addition, an interface on the DR of the PLR is created inthe same subnet to connect to the transit logical switch. The processthen ends.

It should be understood that while the processes 1100 and 1200illustrate a specific order for performing these various operations,these processes are merely conceptual. In various different embodiments,the management plane may perform the actual operations in variousdifferent orders, or even perform some of the operations in parallel.For instance, the management plane could define the transit logicalswitch first, prior to defining the SR or DR at all, could define all ofthe logical router components completely before assigning them toseparate physical machines, etc.

III. Packet Processing

The above sections describe the configuration of the various logicalrouter components by the management plane. These logical routercomponents (as well as the logical switches, both those defined by theuser and those defined by the management plane for connecting logicalrouter components) are implemented in the datacenter by various managedforwarding elements (MFEs). As shown in FIG. 5 , for example, the datacompute nodes attached to the user-defined logical switches reside onphysical host machines, on which MFEs operate (e.g., within thevirtualization software of the host machine) as first-hop packetprocessing elements. These MFEs implement the logical switches of alogical network as well as the DRs, in some embodiments.

FIG. 13 conceptually illustrates a physical implementation of themanagement plane constructs for a two-tiered logical network shown inFIG. 8 , in which the TLR 610 and the PLR 605 both include SRs as wellas a DR. It should be understood that this figure only shows theimplementation of the TLR 610, and not the numerous other TLRs, whichmight be implemented on numerous other host machines, and the SRs ofwhich might be implemented on other gateway machines.

This figure assumes that there are two VMs attached to each of the twological switches 625 and 630, which reside on the four physical hostmachines 1305-1320. Each of these host machines includes a MFE 1325.These MFEs may be flow-based forwarding elements (e.g., Open vSwitch) orcode-based forwarding elements (e.g., ESX), or a combination of the two,in various different embodiments. These different types of forwardingelements implement the various logical forwarding elements differently,but in each case they execute a pipeline for each logical forwardingelement that may be required to process a packet.

Thus, as shown in FIG. 13 , the MFEs 1325 on the physical host machinesinclude configuration to implement both logical switches 625 and 630(LSA and LSB), the DR 805 and transit logical switch 815 for the TLR610, and the DR 705 and transit logical switch 725 for the PLR 605. Someembodiments, however, only implement the distributed components of thePLR on the host machine MFEs 1325 (those that couple to the data computenodes) when the TLR for a data compute node residing on the host machinedoes not have a centralized component (i.e., SRs). As discussed below,northbound packets sent from the VMs to the external network will beprocessed by their local (first-hop) MFE, until a transit logical switchpipeline specifies to send the packet to a SR. If that first SR is partof the TLR, then the first-hop MFE will not perform any PLR processing,and therefore the PLR pipeline configuration need not be pushed to theseMFEs by the centralized controller(s). However, because of thepossibility that one of the TLRs 615-620 may not have a centralizedcomponent, some embodiments always push the distributed aspects of thePLR (the DR and the transit LS) to all of the MFEs. Other embodimentsonly push the configuration for the PLR pipelines to the MFEs that arealso receiving configuration for the fully distributed TLRs (thosewithout any SRs).

In addition, the physical implementation shown in FIG. 13 includes fourphysical gateway machines 1330-1345 (also called edge nodes, in someembodiments) to which the SRs of the PLR 605 and the TLR 610 areassigned. In this case, the administrators that configured the PLR 605and the TLR 610 selected the same group of physical gateway machines forthe SRs, and the management plane assigned one of the SRs for both ofthese logical routers to the third gateway machine 1340. As shown, thethree SRs 710-720 for the PLR 605 are each assigned to different gatewaymachines 1330-1340, while the two SRs 810 and 815 for the TLR 610 arealso each assigned to different gateway machines 1340 and 1345.

This figure shows the SRs as separate from the MFEs 1350 that operate onthe gateway machines. As indicated above, different embodiments mayimplement the SRs differently. Some embodiments implement the SRs as VMs(e.g., when the MFE is a virtual switch integrated into thevirtualization software of the gateway machine, in which case the SRprocessing is performed outside of the MFE. On the other hand, someembodiments implement the SRs as VRFs within the MFE datapath (when theMFE uses DPDK for the datapath processing). In either case, the MFEtreats the SR as part of the datapath, but in the case of the SR being aVM (or other data compute node), sends the packet to the separate SR forprocessing by the SR pipeline (which may include the performance ofvarious services). As with the MFEs 1325 on the host machines, the MFEs1350 of some embodiments are configured to perform all of thedistributed processing components of the logical network.

A. Single-Tier Topology

The packet processing pipelines for various examples will now bedescribed. FIGS. 14A and 14B illustrate examples of traffic thategresses from the logical network (northbound traffic) and ingresses tothe logical network (southbound traffic), respectively, for a logicaltopology with a single tier of logical routers. These figures illustratea single tier topology 1400 with a logical router 1405 (with aconnection to external networks) and two logical switches 1410 and 1415.As described above, the logical router 1405 includes a DR 1420, two SRs1425 and 1430, and a transit logical switch 1435.

In some embodiments, east-west traffic (i.e., traffic from a datacompute node on LS1 to a data compute node on LS2 is handled primarilyat the first-hop MFE (e.g., the MFE of the virtualization software onthe host machine for the source data compute node), then tunneled to thedestination MFE. As such, the packets do not pass through the SRs, andthus does not receive any services provided by these SRs. Otherembodiments, however, allow for routing policies that send certaineast-west traffic to the SRs for processing.

As shown in FIG. 14A, when a VM or other data compute node on a machinesends a northbound packet, the datapath on the MFE initially runs thesource logical switch pipeline (e.g., based on the ingress port throughwhich the packet is received, the source MAC address, etc.). Thispipeline specifies to forward the packet to the DR 1420, the pipelinefor which also takes place on the source MFE. This pipeline identifiesone of the SRs 1425 and 1430 as its next hop. In the active-standbycase, the pipeline identifies the active SR; in the active-active case,some embodiments use ECMP to select one of the SRs, as described below.Next, the source MFE executes the pipeline for the transit logicalswitch 1435, which specifies to tunnel the packet to the appropriategateway machine (edge node) that hosts the selected SR. The gatewaymachine (e.g., the MFE on the gateway machine) receives the packet,decapsulates it (to remove the tunneling data), and identifies the SRbased on the logical context information on the packet (e.g., the VNI ofthe transit logical switch 1435) as well as the destination MAC addressthat corresponds to the SR's southbound interface. The SR pipeline isthen executed (by the MFE in some embodiments, and by a VM implementingthe SR in other embodiments). The SR pipeline sends the packet to thephysical network. If the SR pipeline specifies a local interface, thenthe packet is delivered directly to the physical network; on the otherhand, if the SR pipeline specifies a dummy interface, the packet may beredirected through a tunnel to a different gateway machine to which thespecified interface is local.

FIG. 14B illustrates the packet processing for ingressing (southbound)packets. The packet is received at one of the gateway machines on whichan SR operates. The MFE at the gateway machine identifies thedestination SR based on the VLAN and destination MAC address of theincoming packet, and runs the SR pipeline (e.g., sends the packet to theVM on which the SR operates, or runs the pipeline directly in thedatapath, depending on how the SR is implemented). The SR pipelineidentifies the DR 1420 as its next hop. The MFE then executes thetransit logical switch 1435 pipeline, which forwards the packet to theDR, as well as the DR pipeline, which routes the packet to itsdestination. The destination logical switch pipeline (i.e., one of thelogical switches 1410 and 1415) is also executed, which specifies totunnel the packet to the MFE of the host machine on which thedestination VM resides. After decapsulating the packet, the destinationMFE delivers the packet to the VM.

B. Two-Tier Topology without Centralized Services in TLR

FIGS. 15A and 15B illustrate examples of northbound and southboundtraffic for a two-tier logical topology, with no centralized servicesprovided in the lower (TLR) tier. These figures illustrate a two-tiertopology 1500 with a PLR 1505 (with two uplinks to external networks), aTLR 1510, and two logical switches 1515 and 1520. The PLR 1505 includesa DR 1525, two SRs 1530 and 1535, and transit logical switch 1540 thatconnects the three components. The TLR 1510 does not have centralizedservices configured, and therefore only includes a single DR component1545. Between the DR 1545 of the TLR and the DR 1525 of the PLR themanagement plane inserts a second transit logical switch 1550.

The processing pipeline for the two-tier topology without statefulservices at the TLR level is similar to the single-tier topologypipeline, but with additional pipelines executed at the first-hop MFE.As shown in FIG. 15A, when a VM or other data compute node on a machinesends a northbound packet, the datapath on the MFE of the source machineinitially runs the source logical switch pipeline (e.g., based on theingress port through which the packet is received, the source MACaddress, etc.). This pipeline specifies to forward the packet to the DR1545 of the TLR 1510, the pipeline for which is also executed on thesource (first-hop) MFE. This pipeline identifies the southboundinterface of the DR 1525 as its next-hop, and the source MFE thenexecutes the pipeline for the transit logical switch 1550 interposedbetween the two DRs. This logical switch pipeline logically forwards thepacket to the DR port (the upper-layer DR), and the source MFE thenexecutes the pipeline for the DR 1525 as well. This pipeline identifiesone of the SRs 1530 and 1535 as the next hop for the packet. In theactive-standby case, the pipeline identifies the active SR; in theactive-active case, some embodiments use ECMP to select one of the SRs,as described below.

Next, the source MFE executes the pipeline for the transit logicalswitch 1540 internal to the PLR 1505, which specifies to tunnel thepacket to the appropriate gateway machine (edge node) that hosts theselected SR (identified by the transit logical switch pipeline based onMAC address, in some embodiments). The gateway machine (e.g., the MFE onthe gateway machine) receives the packet, decapsulates it (to remove thetunneling encapsulation), and identifies the SR based on the logicalcontext information on the packet (e.g., the VNI of the transit logicalswitch 1540) as well as the destination MAC address that corresponds tothe SR's southbound interface. The SR pipeline is then executed (by theMFE in some embodiments, and by a VM implementing the SR in otherembodiments). The SR pipeline sends the packet to the physical network.If the SR pipeline specifies a local interface, then the packet isdelivered directly to the physical network; on the other hand, if the SRpipeline specifies a dummy interface, the packet may be redirectedthrough a tunnel to a different gateway machine to which the specifiedinterface is local.

Southbound traffic is also handled similarly to the single-tier case. Asshown in FIG. 15B, a southbound packet is received at one of the gatewaymachines on which an SR of the PLR 1505 operates. The MFE at the gatewaymachine identifies the destination SR (some embodiments allow thegateway machines to host numerous SRs for various different logicalrouters) based on the VLAN and destination MAC address of the incomingpacket, and runs the SR pipeline (e.g., sends the packet to the VM onwhich the SR operates, or runs the pipeline directly in the datapath,depending on how the SR is implemented). The SR pipeline identifies theDR 1525 as its next hop, so the MFE then executes the transit logicalswitch 1540 pipeline, which forwards the packet to the DR 1525. The DR1525 pipeline identifies the TLR DR 1545 as its next hop, and thus theMFE on the edge node also executes the pipeline of the transit logicalswitch 1550 and subsequently, that of the DR 1545. The lower-level DRpipeline routes the packet to its destination, so the destinationlogical switch pipeline (i.e., one of the logical switches 1515 and1520) is also executed, which specifies to tunnel the packet to the MFEof the host machine on which the destination VM resides. Afterdecapsulating the packet, the destination MFE delivers the packet to theVM.

For east-west traffic, in some embodiments, the source MFE handles allof the processing, as in the single-tier case. Within a TLR (e.g., froma VM on the first logical switch 1513 to a VM on the logical switch1520, only the single DR pipeline (and the two logical switch pipelines)needs to be executed. For packets sent across TLRs, the source MFEexecutes all three of the DR pipelines in some embodiments (so long asthe destination TLR-DR and logical switch pipelines are implemented onthe source MFE. As in the single-tier case, some embodiments alloweast-west traffic to be sent to the SRs on the gateway machines, whileother embodiments do not enable the centralized services for east-westtraffic.

C. Two-Tier Topology with Centralized Services in TLR

Finally, FIGS. 16A and 16B illustrate examples of northbound andsouthbound traffic for a two-tier logical topology with centralizedservices provided in the lower (TLR) tier by SRs. These figuresillustrate a two-tier topology 1600 with a PLR 1605 (with two uplinks toexternal networks), a TLR 1610 (with centralized services), and twological switches 1615 and 1620. The PLR 1605 includes a DR 1625, two SRs1630 and 1635, and a transit logical switch 1640 that connects the threecomponents. The TLR also includes a DR 1645, two SRs 1650 and 1655, anda transit logical switch 1660 that connects its three components. Themanagement plane also has inserted a third transit logical switch 1665between the SRs 1650 and 1655 of the TLR 1610 and the DR 1625 of the PLR1605.

Unlike the previous examples, in which nearly the entire packetprocessing pipeline was performed at the first hop, packet processingfor the logical topology 1600 is spread across three machines for bothnorthbound and southbound traffic. As shown in FIG. 16A, when a VM orother data compute node on a machine sends a northbound packet, thedatapath on the MFE of the source machine initially runs the sourcelogical switch pipeline, as in the previous examples. This pipelinespecifies to forward the packet to the DR 1645 of the TLR 1610, thepipeline for which is also executed on the source (first-hop) MFE. ThisDR pipeline identifies the southbound interface of one of the SRs 1650and 1655 as its next hop IP address. In some embodiments, the TLR SRsare always configured in active-standby mode, so the next hop is thesame for both of the SRs but the packet is routed to the MAC address ofthe active SR.

The source MFE then executes the pipeline for the transit logical switch1660 internal to the TLR 1610, which specifies to tunnel the packet tothe appropriate gateway machine (edge node) that hosts the selected SRof the TLR 1610 (which the transit logical switch identifies based onthe destination MAC address after routing by the DR 1645 pipeline). Thegateway machine (e.g., the MFE on the gateway machine) receives thepacket, decapsulates it, and identifies the SR based on the logicalcontext information on the packet (e.g., the VNI of the transit logicalswitch 1660) as well as the destination MAC address that corresponds tothe SR's southbound interface. The SR pipeline (including any of thestateful services) is then executed (e.g., by the MFE or a VMimplementing the SR), which specifies the southbound interface of the DR1625 as its next hop address. The transit logical switch 1665 pipelineis executed on the current edge node (Edge Node 2 in the figure), as isthe DR pipeline of the PLR 1605. This DR pipeline identifies one of theSRs 1630 and 1635 as the next hop for the packet, in the same manner asdescribed in the previous examples.

The edge node MFE executes the pipeline for the transit logical switch1640 internal to the PLR 1605, which specifies to tunnel the packet tothe appropriate gateway machine that hosts the selected SR 1630 or 1635(identified by the transit logical switch pipeline based on MAC address,in some embodiments). The gateway machine (e.g., the MFE on the gatewaymachine) receives the packet, decapsulates it (to remove the tunnelingencapsulation), and identifies the SR based on the logical contextinformation on the packet (e.g., the VNI of the transit logical switch1640) as well as the destination MAC address that corresponds to theSR's southbound interface. The SR pipeline is then executed (by the MFEin some embodiments, and by a VM implementing the SR in otherembodiments). The SR pipeline sends the packet to the physical network.If the SR pipeline specifies a local interface, then the packet isdelivered directly to the physical network; on the other hand, if the SRpipeline specifies a dummy interface, the packet may be redirectedthrough a tunnel to a different gateway machine to which the specifiedinterface is local.

Southbound traffic processing is also distributed across three machines(unless the SR for the PLR 1605 and the SR for the TLR 1610 are locatedon the same gateway machine). As shown in FIG. 16B, a southbound packetis received at one of the gateway machines on which an SR of the PLR1605 operates. The MFE at the gateway machine identifies the destinationSR based on the VLAN and destination MAC address of the incoming packet,and runs the SR pipeline (e.g., sends the packet to the VM on which theSR operates, or runs the pipeline directly in the datapath, depending onhow the SR is implemented). The SR pipeline identifies the DR 1625 asits next hop, so the MFE then executes the transit logical switch 1640pipeline, which forwards the packet to the DR 1625. The DR 1625 pipelineidentifies the northbound interface of one of the SRs 1650 and 1655 ofthe TLR 1610 as its next hop. In the active-standby case, the active SRis selected.

The MFE on the first gateway machine then executes the transit logicalswitch 1665 pipeline, which specifies to tunnel the packet to a secondgateway machine (Edge Node 2) on which this second SR that performsstateful services for the TLR 1610 is located. The second gatewaymachine (e.g., the MFE on the second gateway machine) decapsulates thepacket and identifies the destination SR based on the VNI and MACaddress on the packet. The MFE runs the SR pipeline (either in itsdatapath or by sending the packet to a VM on the gateway machine), whichidentifies the DR 1645 as the next hop. The MFE thus executes thetransit logical switch 1660 pipeline, which forwards the packet to theDR 1645, and then executes this DR pipeline as well. The DR pipelineroutes the packet to its destination, so the destination logical switchpipeline (one of the logical switches 1615 and 1620) is executed, andthe packet is tunneled to the MFE of the host machine on which thedestination VM resides. After decapsulating the packet, the destinationMFE delivers the packet to the VM.

For east-west traffic within a TLR, the source logical switch, DR, anddestination logical switch pipelines are all executed at the first-hopMFE, then the packet is tunneled to the destination MFE. IF the packetrequires processing by the centralized services, only the source logicalswitch, DR, and transit logical switch pipelines are performed at thefirst-hop MFE, with the SR pipeline, transit logical switch (again), DR(again), and destination logical switch pipelines performed by thegateway machine before tunneling the packet to the destination. Forcross-TLR traffic, the packet starts out in the same way, with thefirst-hop MFE performing the source logical switch, DR, and transitlogical switch pipelines to select a SR. The gateway machine on whichthe selected SR runs then executes the SR pipeline to identify the DR ofthe PLR, the transit logical switch pipeline between the TLR and thePLR, the DR of the PLR pipeline (which identifies a next hop as acomponent of a different TLR), and at least the transit logical switchbetween the PLR and the destination TLR. If the destination TLR has onlya DR, then that pipeline is also executed at the first gateway machine,along with the destination logical switch, before tunneling the packetto its destination MFE. If the destination TLR has SRs, the transitlogical switch specifies to tunnel the packet to the gateway machine fora selected SR of the destination TLR. That second gateway machineexecutes the SR pipeline, the transit logical switch pipeline internalto the destination TLR, the DR pipeline for that TLR, and thedestination logical switch pipeline, before tunneling the packet to thedestination MFE.

The same principle applies in all of the above cases, which is toperform the processing pipelines as early as possible. Thus, all of thepipelines for a given packet are performed at the first-hop MFE (e.g.,the hypervisor-based virtual switch that receives a packet from a VM onthat hypervisor), until the packet needs to be sent to a SR pipelineonly present on a specific gateway machine. That gateway machine thenperforms all of the processing it can, until the packet is sent out to aphysical network or to a different gateway machine (or to itsdestination for east-west traffic).

D. Additional Logical Router Behavior

Much like physical routers, logical routers are implemented to performtypical routing functionalities such as decrementing the time to live(TTL) for packets that it routes, and performing ARP. In someembodiments, a logical router with both DR and SRs only decrements apacket once, by the first component that acts upon the packet. Thus, fornorthbound and east-west traffic, the DR decrements the TTL, whereas theSR decrements the TTL for southbound traffic. In some embodiments, theDR implementation has instructions to only decrement TTL for packetsreceived on its southbound interface, and the SRs have similarinstructions to only decrement TTL for packets received on theirnorthbound interfaces. The component that handles decrementing the TTLfor a packet also handles generating an ICMP error message if the TTL isdropped to zero.

The logical routers of some embodiments do not forward broadcastpackets, and thus do not support directed broadcast (a feature typicallydisabled on physical routers as well). However, if an IP broadcastpacket is received on the logical network to which it is addressed, thelogical router of some embodiments treats itself as a destination of thepacket.

For ARP, in some embodiments, the logical router rewrites the MACaddress of the inner packet (i.e., the packet before a tunnelencapsulation is appended to the packet) to indicate which transportnode is sending the ARP packet, so that the ARP response is forwarded tothe correct transport node. For the tunnel encapsulation, someembodiments use stateless transport tunneling (STT) along with VXLANsemantics.

E. Packet Processing by SR

The above descriptions describe the packet processing by the SR assimply one additional logical forwarding element in the datapath for apacket, which may not be implemented at the first hop (for northbound oreast-west packets, at least). However, where the other logicalforwarding elements (logical switches, DRs, transit logical switches)basically involve ingress processing, logical forwarding, and egressprocessing (the ingress and egress processing may involve ACLs), the SRprocessing may include other functions such as stateful services inaddition to the forwarding-related processing.

FIG. 17 conceptually illustrates the various stages of SR processing1700 of some embodiments. Some of these stages are only included in theprocessing when the SR includes non-forwarding services (e.g., NAT,stateful firewall, load balancing, etc.). Thus, the diagram showscertain stages in dashed rather than solid lines to indicate that the SRonly performs these stages if configured for services. In addition, thepre-service redirect stage 1705 is illustrated using dotted lines toindicate that the SR only performs this stage if the SR containsservices and its logical router is configured in active-active mode.

As shown, when a SR receives a packet (whether the SR is implemented asa VM or as a VRF in a DPDK-based datapath), the first stage 1705 is thepre-service redirect operation. As mentioned, the SR only performs thisstage if stateful services are configured and the SRs are operating inactive-active mode. The pre-service redirect stage 1705 involvesredirecting the packet to the owner SR for a connection (e.g., atransport connection) to which the packet belongs. However, if noservices are configured on the logical router, or the SR is operating inactive-standby mode (in which case all packets are sent to the activeSR), then this stage is not needed. In some embodiments, the pre-serviceredirect stage does not decrement TTL (as the packet will be properlydecremented when routed at a later stage).

The pre-routing service stages 1710-1715 may involve any number ofstateful services configured on the SR for performance prior to routing.The SR performs these stages upon determining that no redirect isnecessary or receiving a packet via redirect from a different SR. Ofcourse, if no stateful services are configured on the SR, theseoperations will not be performed as well. Depending on the configurationof the SR, and whether certain services require the determination of anegress logical port of the logical router, some services may beperformed either before or after routing.

After all the pre-routing services have been performed by the SR, the SRthen performs the routing stage 1720. As discussed above, the routingtables for all of the SR instances will be similar. For instance, ifmultiple SRs can reach the same network, then all SRs will have multipleroutes for that network, with routes that point to a local interfacehaving a smaller distance metric than routes that point to a dummyinterface, so the local interface will be chosen when possible. Therouting stage 1720 results in a routing decision, which includes a nexthop IP address and an egress logical port of the logical router (in someembodiments, the egress logical port may already be known based onrouting performed by the DR for northbound packets).

After being routed, the packet proceeds to the post-routing servicesstages 1725-1730. These stages, like the pre-routing services stages1710-1715, are only performed by the SR if stateful services areconfigured on the logical router. In some embodiments, some or all ofthe post-routing service stages may depend on the routing decision. Forexample, interface-based NAT configured for the logical router maydepend on the logical egress port. In addition, some embodiments requirethat the post-routing services do not alter the routing decision (thoughthey may cause the SR to drop the packet, in some cases).

Next, the SR processes the packet through the egress ACL stage 1735. Atthis stage, the SR enforces any security policies configured for thelogical egress port of the logical router. The SR then ARPs (at stage1740) the next hop to determine the new destination MAC address for thepacket. When the egress interface of the SR is a dummy interface, insome embodiments the ARP is injected into the destination L2 via proxyin the same way that the DR performs ARP in the logical space. After ARPconcludes, the SR modifies the source and destination MAC addresses ofthe packet.

Lastly, the packet proceeds to the egress output stage 1745. If theegress interface is local, the packet is sent to the proper VLAN. On theother hand, if the egress interface is remote, the SR forwards thepacket to the dummy interface's SR, which then sends the packet out viathe proper VLAN. In some embodiments, the packet is sent to the correctpeer SR, which then performs ARP and outputs the packet. However, thistechnique requires either for the packet to store next-hop informationor for the peer SR to re-perform the routing stage. In some embodiments,the egress output stage does not decrement TTL. The TTL is insteaddecremented by either the routing stage at this SR or, if receivedthrough redirect at the output stage of a different SR, then by therouting stage at that different SR.

IV. ECMP Routing in Multi-Tier Logical Networks

As mentioned above, some embodiments use equal-cost multi-path routingtechniques, for both northbound and southbound packets, with regard tothe SRs of a PLR. In some embodiments, the use of ECMP is only allowedwhen no stateful service is configured on the logical router thatinterfaces with the physical network (e.g., the PLR in a two-tiertopology). In order for packets to be forwarded using ECMP techniques, aPLR requires multiple uplinks and for BGP (or another dynamic routingprotocol) to be enabled. In some embodiments, the multiple uplinks maybe located in the same L2 domain.

As described previously, the user (administrator) associates a logicalrouter with a particular set of physical gateway machines. Themanagement plane then assigns the various uplinks of the PLR todifferent gateway machines in this set of physical gateway machines.Some embodiments enforce a rule that the various gateway machines withina specifiable set have uniform physical connectivity to the externalnetwork (e.g., that all of the machines have access to the same set ofVLANs), which simplifies the logic at the management plane. At eachgateway machine to which the management plane has assigned an uplink, anSR is created.

Some embodiments place additional requirements on the uniform physicalconnectivity. Specifically, in some embodiments all of the gatewaymachines spanned by a PLR have the same L3 connectivity (i.e., all ofthese machines connect to the same set of physical routers).Furthermore, with BGP enabled (a requirement for ECMP), all of thesephysical next-hops (the physical routers) are required to have the samephysical connectivity. This means that all SRs for a particular PLR willreceive the same set of routes from their physical next-hops, with thepossibility of transient route differences between SRs that disappearfairly quickly. With this set of requirements, the dummy uplinks are notrequired, as packets will not need to be redirected between uplinks (asall uplinks have the same policies and same connectivity).

FIGS. 18 and 19 illustrate a single-tier logical network topology 1800and the management plane view of that topology that meets theabove-stated requirements for the use of ECMP. The network topology 1800is similar to that of FIG. 9 , but each of the two uplinks has the sameL3 connectivity. The logical network topology 1800 includes two logicalswitches 1805 and 1810 that connect to a logical router 1815. Theconfiguration of these components is the same as with the networktopology 900, except for the configuration of the physical routers towhich the uplinks connect. That is, the interfaces between the logicalrouter 1815 and the logical switches 1805 and 1810 are all the same, andthe two uplinks U1 and U2 of the logical router 1815 connect to physicalrouters 1820 and 1825 with the same next hop IP addresses. However,whereas in the previous example the physical routers providedconnectivity to different networks, here the physical routers both havethe same L3 connectivity to the Internet.

Thus, in FIG. 19 , the management plane view 1900 of the logical networkis nearly the same as well. The management plane again defines, for thelogical router 1815, a DR component 1905, two SRs 1910 and 1915 for thetwo uplinks, and a transit logical switch 1920. The only modification tothe configuration is that no dummy interfaces are configured on the SRs,because the two uplinks have the same configuration and RIB, so one ofthe SRs should not receive a packet that needs to be forwarded out ofthe second SR. As such, the routes in the RIB for redirection that weredescribed in the previous section will not be included in the RIB ofthese SRs.

In some embodiments, ECMP is used in conjunction with BGP (or otherdynamic routing protocols). Each SR of the logical router establishes aBGP session with the one or more physical routers to which it connects.For instance, in the example of FIG. 18 and FIG. 19 , the SR 1910initiates a session with the physical router 1820, while the SR 1915initiates a session with the physical router 1825. In some embodiments,each of the uplinks would be connected to both of the physical routers,and thus each uplink would have two routing sessions. In someembodiments, a module on the gateway machine separate from the SRimplementation initiates the BGP session with the router. For instance,when the SR is implemented as a VM, the BGP module may be part of the VMor a separate module operating as part of the hypervisor, in a separateVM or other data compute node, etc. During these sessions, the SRadvertises the prefixes in the logical space (e.g., the logical switchsubnets 1.1.1.0/24 and 1.1.2.0/24) to the physical routers, using thesame metric for each of the prefixes. The BGP integration techniques ofsome embodiments are described in U.S. patent application Ser. No.14/214,561, filed Mar. 14, 2014, which is incorporated herein byreference.

With all of the SRs advertising the same routes to the physical routers,the physical routers can then treat the SRs as equal-cost routingoptions, and spread traffic through the various SRs. In the exampleshown in FIGS. 18 and 19 , each of the physical routers can only sendpackets to one of the SRs. However, each of the physical routers has thesame connectivity, so packets sent from the networks behind them towardsthe logical network will be spread evenly between the two routers 1820and 1825, and therefore spread evenly between the two SRs. When each SRconnects to all of the physical routers, then each of these physicalrouters can spread traffic evenly between the SRs on their own.

For northbound packets, the DR of some embodiments uses ECMP techniquesto distribute packets among the various SRs, which provide equalconnectivity for northbound packets. By running BGP (or a differentdynamic routing protocol), the SRs learn routes from the physicalrouters in addition to advertising routes for the logical networkprefixes. As mentioned, the SRs locally incorporate these routes intotheir RIBs, and can recalculate their FIBs based on the newly learnedroutes. However, for the DR to use ECMP, the routes must also be givento the RIB of the DR, which is implemented at numerous machines.

In some embodiments, the SRs report the learned routes to thecentralized network controllers that configure and manage the SRs (aswell as the MFEs that implement the distributed logical forwardingelements). The centralized controllers then update the RIB of the DRaccordingly, and distribute the updates to the MFEs that implement theDR. Different embodiments may update the DRs at different rates,depending on the desired balance between keeping an up-to-date RIB andthe processing load on the central controllers. Rather than distributingthe RIB, some embodiments compute the FIB at the centralizedcontrollers, then distribute the updated FIB to the MFEs that implementthe DR.

In other embodiments, rather than continuously updating the routes, thecentralized controller instead adds to the DR RIB default routes thatpoint to all of the SRs. These routes are classified as management planeinternal, so they are only used by the DR if they are not overruled bystatic routes input by an administrator. Because the routes for thedifferent SRs have the same administrative distance metric, the DRtreats them as equal-cost options, dividing traffic between the SRs withECMP techniques.

V. Active-Standby for Stateful Services

While the above section describes the SR setup for active-activeconfiguration with ECMP (when all of the two or more SRs are treated asequal options), some embodiments use an active-standby configurationwith two SRs. Some embodiments use the active-standby configuration whenstateful services are configured on the SRs. In this case, the benefitof avoiding having to continuously share state between the SRs mayoutweigh the negatives of sending all of the northbound and southboundtraffic between multiple SRs (while using a standby for backup in caseof failure). In the active-standby case, the state is periodicallysynchronized between the two SRs, though this need not be done at perpacket speeds.

In some embodiments, for active-standby configuration, the administratoris required to configure two uplinks when defining the logical router,and the uplinks need not be in the same L2 domain. However, because theactive and standby SRs should be equivalent options to the DR (with theactive SR the preferred of the two options), some embodiments requirethe two SRs to have uniform L3 connectivity. This is, of course, not anissue when the active-standby SRs are configured for a TLR with statefulservices, as both SRs will have one next hop, the DR of the PLR to whichthe TLR connects. For a PLR in active-standby configuration, the twouplinks should be configured with the same connectivity in someembodiments. In addition, for a PLR, some embodiments allow (or require)the configuration of dynamic routing protocols (e.g., BGP) on the SRs.

FIG. 20 illustrates a management plane view 2000 of the logical networktopology 1800 when the logical router is configured in active-standbymode, rather than active-active (ECMP) mode. Here, the only differencein configuration from the active-active mode shown in FIG. 19 is thatthe southbound interfaces of the SRs 2010 and 2015 are assigned the sameIP address, but different MAC addresses.

The management plane configures the DR 2005 in the same manner as in thegeneral case of FIGS. 9 and 10 , in terms of assigning MAC and IPaddresses to its southbound and northbound interfaces. When constructingthe RIB, the same connected routes are used, and the same static routerules apply as described above in Section II (e.g., northbound routesare copied to the DR but modified to set the SR IP address as its nexthop). In this case, because there is only one IP address for the SR, allnorthbound routes use this single IP as the next hop address. Similarly,rather than creating multiple default routes to the various different SRIP addresses, a single default route with this lone IP address as thenext hop is added to the RIB of the DR. Thus, the RIB for the DR 2005 inFIG. 20 includes the following routes:

-   -   1.1.1.0/24 output to L1    -   1.1.2.0/24 output to L2    -   192.168.100.0/24 output to DRP1    -   192.168.1.0/24 via IP1    -   192.168.2.0/24 via IP1    -   0.0.0.0/0 via IP1

Each of the SRs 2005 will be configured in mostly the same manner. Whenthe logical router is a PLR (or in a one-tier topology, as in theexample), the IP and MAC addresses of the northbound interfaces are thesame as those assigned to the two uplinks configured for the PLR. On theother hand, when the logical router is a TLR, it may only have oneuplink that is configured to connect to the PLR. In this case, the IPaddresses of the two northbound interfaces are the same, but each SR isassigned a different MAC address. Similarly, in either of these twocases (PLR or TLR), a single IP address is assigned to the twosouthbound interfaces (as in FIG. 20 , in which both of these interfaceshave an IP address of IP1), with two different MAC addresses for the twoSRs.

Any uplink-independent service policies the controller pushes to both ofthe SRs identically, in some embodiments. If any service policies thatdepend on the uplink are allowed and configured, then these are pushedto the SRs on which the uplink with which they are associated exists. Inaddition, any dynamic routing configurations of a logical router portare transferred to the northbound interface of the SRs.

The RIB for the SRs is similar to that described above in Section II forthe general case. Static and connected routes that egress from an uplinkof the logical router are added to the RIB of the SR withoutmodification. For each southbound interface of the logical router (e.g.,routes for logical switch subnets), a route for the network is addedwith the next hop IP address set to the northbound interface of the DR.Any route in the RIB of the logical router that egresses from thissouthbound interface is also added to the RIB of the SR with this samenext hop IP address. The RIB of SR1 2010 in the example of FIG. 20 willinclude the following routes, prior to learning any additional routesvia dynamic routing protocols:

-   -   0.0.0.0/0 output to U1 via 192.168.1.252    -   192.168.1.0/24 output to U1    -   192.168.100.0/24 output to SRP1    -   1.1.1.0/24 via IP3    -   1.1.2.0/24 via IP3

In addition, when the SR is set as a standby SR (rather than active SR),the SR does not answer ARP on its southbound interface in someembodiments. ARP packets for the southbound IP of the SR will bebroadcast on the transit logical switch that connects the SRs and theDR, and both the active and standby SRs will be responsive to that IPaddress. However, only the active SR will respond to ARP requests, sothat the DR will route packets to the MAC address of the active SRrather than the standby SR. The standby SR in some embodiments willnevertheless accept packets received by the northbound interface, inorder to run its dynamic routing protocol and keep an up-to-date set ofroutes in case it becomes the active SR. However, the standby SR doesnot advertise prefixes to the external networks, unless it becomesactive.

VI. SR Failover

As described above, the SRs may be implemented in different embodimentsas VMs or other data compute nodes or as VRFs within DPDK-baseddatapaths. In both cases, the possibility of different types of failure(partial tunnel failure, complete tunnel failure, physical machinecrashes, etc.) may cause a SR to go down. However, different SRimplementations may respond to different types of failures in differentmanners.

A. Failure Handling with DPDK-Based SRs

In some embodiments, as described, the SRs of a logical router operateon gateway machines, or edge nodes, as VRFs within the DPDK-baseddatapaths. These gateway machines are grouped into sets (e.g., based onphysical location within a datacenter), and the gateway machines of aset that collectively host all of the SRs for a particular logicalrouter are connected by a set of tunnels (e.g., a full mesh of tunnelsin some embodiments). Thus, tunnels exist between all of the gatewaymachines on which a SR operates.

Some embodiments use Bidirectional Forwarding Detection (BFD) sessionsto maintain these tunnels, in order to monitor the aliveness of peergateway machines. However, as using only the single BFD session betweenthe tunnel endpoints would require depending on a single informationchannel to detect the aliveness of a peer, some embodiments also use asecond channel between each pair of gateway machines. Specifically, insome embodiments, a separate management network exists between thegateways for sending control data (e.g., for communication with thenetwork controllers). Thus, each gateway has a separate IP address onthe management network, and these connections may be used to sendheartbeat messages over the management network. This prevents thepossibility of tunnel failure between two peers resulting in both of thegateway machines determining that the other has crashed and initiatingactions that cause confusion when the peer is not actually down.Instead, during tunnel failure, each of the nodes can detect that theirpeer machine is still up, and thus conclude that the tunnel has failedand not the peer machine (and thus its SRs) itself.

In some embodiments, the failure conditions are different for SRs ofPLRs and SRs of TLRs. When the tunnels of a gateway machine that provideconnectivity to the MFEs on which the user VMs run (e.g., the MFEs 1325of FIG. 13 to which the user VMs directly connect) fail, all SRs on thegateway machine are no longer operational (even for the SRs of PLRs, astraffic sent to the PLRs by external physical routers will beblackholed. On the other hand, when a gateway machine loses itsconnectivity to the physical routers, the SRs of TLRs on the gateway arestill treated as operational, as northbound traffic to the TLRs willhave the DR of a PLR as a next hop, which should always be available (asit is also implemented within the datapath on the gateway). The SRs ofPLRs, however, are no longer considered operational, as any northboundtraffic originating from VMs of the logical network will be blackholed.When a gateway machine that hosts SRs of PLRs loses its physicalconnectivity (or its BGP sessions), in some embodiments the gatewaymachine sends a message (e.g., a specific diagnostic code such as“concatenated path down”) to other gateway machines that host SRs of thesame PLR.

Based on the BFD session on the tunnel with a peer, the status ofheartbeat messages over the second (e.g., management) channel with thepeer, and whether a message has been received from the peer indicatingthat the peer's physical connectivity is down, a first gateway machinecan make a conclusion about its peer second gateway machine and takecertain actions based on those conclusions. For example, if the tunnelis active and no connectivity down message is received, then the firstgateway machine concludes that the peer second gateway machine ishealthy, and continues processing packets as normal. However, if thetunnel to the peer is up, but the connectivity down message has beenreceived, then the first gateway machine concludes that the peer isstill active but has lost its physical connectivity. As such, the SR onthe first gateway machine takes over the SR (as described below) on thesecond gateway machine if the SR belongs to a PLR, but takes no actionwith regard to SRs of TLRs.

If the tunnel goes down (based on the BFD session no longer beingactive) between the first gateway machine and the peer second gatewaymachine, but the secondary channel heartbeat messages are stillreceived, then the first gateway machine concludes that the peer secondgateway machine is still healthy and handling northbound and southboundpackets (although redirection may be a problem if needed). However, ifboth the tunnel and the secondary channel are down, then the firstgateway machine concludes that the peer has gone down (e.g., crashed).In this case, the SR on the first gateway machine takes over for the SRon the second gateway machine (as described below), irrespective ofwhether the SRs belong to a PLR or a TLR.

In some embodiments, each gateway machine has a local network controller(sometimes referred to as a chassis controller) that operates on themachine. The chassis controller of some embodiments receives data tuplesfrom the central network controller and uses the data tuples toconfigure the MFE on the machine. This chassis controller is also, insome embodiments, responsible for determining when the health status ofits gateway machine changes, as well as when that of a peer gatewaymachine changes. When one of the three indicators of communication(tunnel BFD session, secondary channel, and physical connectivity downmessages) between the gateway machines is affected (based on a loss ofconnectivity, the gateway machine crashing, etc.), the chassiscontroller of some embodiments determines how this affects each SRhosted on its gateway machine.

The actions taken by the chassis controller with respect to a particularone of its SRs then depend on (i) whether the SR belongs to a PLR or aTLR, (ii) whether the SR works in active-active or active-standby mode,(iii) its own local health status, and (iv) the health status of thepeer gateway machine(s) hosting the other SRs of the same logicalrouter. For example, the chassis controller could determine that itslocal SR should no longer be treated as functional, in which case it maysend signals to this effect to a combination of (i) other gatewaymachines, (ii) the host machines on which user VMs reside, and (iii)physical external routers. The chassis controller can also make thedetermination that a local SR should become active, in which case it maystart a failover process to activate the SR. Furthermore, the chassiscontroller could make the determination that a remote SR is no longerfunctional, and start a failover procedure to take over this remote SRlocally.

When a failure condition is detected, various embodiments may takevarious different actions to partially or completely remedy thesituation. Different types of failure cases may include complete orpartial tunnel failure, gateway machine or MFE crashes, link aggregategroup (LAG) status going down, BGP session failing, non-uniform routesamong SRs. While resurrection of an SR is not actually a failurescenario, it also results in actions taken by the gateway machinechassis controller(s) to manage the SRs.

1. Complete Tunnel Failure

Complete tunnel failure may occur due to the gateway machine crashing,or due to pNIC or physical network issues. When complete tunnel failureoccurs at a particular gateway machine, (i) all of the MFEs at hostmachines with user VMs or other data compute nodes lose tunnels to theparticular gateway machine, (ii) other gateway machines lose tunnels tothe particular gateway machine, and (iii) the particular gateway machineloses tunnels to the other gateway machines.

From the point of view of the MFE at a host machine, when its tunnel tothe particular gateway machine fails, the DR of a PLR can reach some SRs(assuming all of the gateway machines spanned by the PLR do not fail atonce) but cannot reach the SR on the particular gateway machine. Assuch, in some embodiments, the datapath or chassis controller on thehost machine automatically removes the affected routes (that use the SRon the particular gateway machine as the next hop IP address) from theFIB of the DR. Some embodiments associate each next hop with a virtualtunnel endpoint (VTEP) of the respective gateway machine. When thetunnel towards a particular VTEP is down, all next hops associated withthe particular VTEP are marked as down, and thus removed whencalculating the FIB for the DR by the local chassis controller.

The other gateway machines detect the failure of the particular gatewaymachine tunnels through the status of the BFD sessions, and that thesecondary channel is still up. These other gateway machines (e.g., thelocal chassis controller on the other gateway machines) can theninitiate a failover process to take over the SRs hosted on the failedgateway machine.

For SRs on the failed gateway machine that are configured inactive-active mode, some embodiments use a ranking mechanism todetermine how the failed SR is taken over by one of the other machines.In some embodiments, the management plane assigns each of the N SRs inan active-active configuration a ranking, from 1 to N. These rankingsmay be assigned randomly, or using a different technique, and aredistributed to the local chassis controller of all of the gatewaymachines that host SRs for a particular logical router in active-activeconfiguration. Based on the ranking of the failed SR, the next-highestranked SR automatically takes over the southbound interface of thefailed SR. For the northbound interface, no action needs to be taken bythe other SRs, as the physical routers will recognize that the SR isdown when the BGP session terminates. To take over the interface, theovertaking SR sends several gratuitous ARP (GARP) messages for all ofthe IP addresses that it is taking over to the transit logical switch onits southbound interface. These messages announce that the IP addressesare now associated with the MAC address of its southbound interface. Ifthe failed SR has already taken over other IP addresses (due to previousfailure of other SRs for the logical router), then multiple IP addressesare taken over by the new overtaking SR.

For SRs on the failed gateway machine that are configured inactive-standby mode, some embodiments treat the failure of the active SRand the failure of the standby SR differently. Specifically, if thefailed SR is a standby, some embodiments take no action (i.e., they donot instantiate a new standby machine), on the assumption that thestandby machine will be brought back up in good time. If the failed SRis the active SR of a TLR, then both the southbound and northboundinterface IP addresses are migrated to the standby SR. Because the TLRhas only a single uplink, both of the SRs share both northbound andsouthbound IP addresses, but with different MAC addresses. In bothcases, some embodiments send GARP messages to the relevant transitlogical switch to effectuate the migration of the IP addresses. For theSR of a PLR, only the southbound interface is migrated, because the twouplinks should have separate IP addresses even in active-standby mode.Furthermore, the new active SR begins advertising prefixes to physicalrouters to draw southbound packets to itself rather than to the failedSR. In the case in which the same IP and MAC addresses are used for thesouthbound interfaces of the active-standby SRs, some embodiments useReverse ARP (RARP) to refresh the MAC:VTEP mapping (that is, so packetswill be sent over the correct tunnel to the newly active SR).

On the gateway machine that has lost all of its tunnels, the chassiscontroller determines that the most likely cause is some sort of localfailure, and thus determines that its local SRs should no longer beactive. Thus, any SR that is announcing prefixes to the externalphysical routers via BGP session withdraws its announced prefixes, so asto avoid attracting southbound traffic that will be blackholed.

2. Partial Tunnel Failure

Partial tunnel failure occurs when only some of the tunnels between thegateway machine and other machines in the datacenter go down. This couldbe due to complete failure at one of the machines with a tunnel to theparticular gateway machine (which would result in the loss of onetunnel), due to conditions at the particular gateway machine that resultin some of its tunnels going down, etc. Described here is the case whenconditions at the particular gateway machine result in a subset of itstunnels failing. As a result, (i) some of the MFEs at host machines withuser VMs or other data compute nodes lose tunnels to the particulargateway machine, (ii) some of the other gateway machines lose tunnels tothe particular gateway machine, and (iii) the particular gateway machineloses tunnels to some other gateway machines.

The MFEs at host machines that lose tunnels to the particular gatewaymachine treat this in the same manner as complete tunnel failure, asfrom the perspective of the host machine this is simply an unreachablegateway. As such, the datapath or chassis controller on the host machineautomatically removes the affected routes that use the SR on theparticular gateway machine as the next hop IP address from the FIB ofthe DR, as described above in subsection 1.

As noted, partial tunnel failure can result in various differentscenarios. For instance, in some cases, a gateway machine may bereachable by some of the host machine MFEs, but not by its peers.Referring to FIG. 13 (which illustrates SRs as VMs but is neverthelessapplicable) as an example, the gateway machine 1330 might be reachableby the host machines 1305-1320 but not reachable by gateways 1335 and1340. In this case, the local chassis controller on the gateway machine1330 will take over the SRs of the PLR that are running on both gatewaymachines 1335 and 1340. In addition, the gateway machine 1335 (ormachine 1340, depending on the ranking) will take over the SR running onthe gateway machine 1330. This results in some of the MFEs (that canreach all of the gateway machines) receiving replies from multiplegateway machines when the DR running on it sends an ARP request for thesouthbound interface IP address of the SR hosted on the first gatewaymachine 1330. So long as the SRs are in an active-active configuration(with no stateful services), this will not create a correctness problem.However, in the case of an active-standby configuration, this would meanthat both of the SRs are now active, which could cause trafficdisruption issues.

Partial tunnel failure can also cause problems in active-standby modewhen, at a particular gateway machine, the tunnels to some of the hostmachines go down, but the peer gateway machines remain reachable. Inthis case, because the tunnels between the SRs are functioning, nofailover occurs. In active-active mode, the datapath at the hostmachines (or the local chassis controller) can make the decision toforward traffic over the tunnels that are still up without issue.However, in active-standby mode, if the tunnels to the active SR aredown, then the MFE will send packets to the standby SR, which does notprocess them. Similarly, in both active-active and active-standbyconfigurations, the gateway machine may not be able to pass onsouthbound traffic from physical routers, which is therefore blackholedin some embodiments.

3. Machine Crash

In some cases, the entire gateway machine may crash, or the DPDKfastpath may crash. As the fastpath is responsible for sending the BFDpackets in some embodiments, either of these situations is the same as acomplete tunnel failure. As the MSR process (which handles BGP sessionsfor the SRs on the gateway machine) may continue to run when only thefastpath crashes (and not the entire gateway machine), physical routerswill still have the ability to send packets to the gateway machine. Thistraffic is blackholed in some embodiments until the fastpath isrestarted.

4. LAG Status Down

In some embodiments, the gateway machines use link aggregate groups(LAG) to reach the external physical routers. When a gateway machinethat hosts a SR of a PLR loses the entire LAG, in some embodiments themachine sends the physical connectivity down message (described above)over tunnels to its peer gateway machines that also host the SRs of thatPLR. In this case, the takeover procedure described above with respectto complete tunnel failure occurs (the next highest-ranked SR takes overthe IP addresses of the SR).

Some embodiments instead mark all tunnels as down as a technique toinduce failover. However, this results in the SRs of TLRs on the machinebeing failed over to other gateway machines as well, which isunnecessary when only the physical connectivity is down. This can leadto numerous GARP messages sent to the MFEs at host machines, andtherefore some embodiments use the first technique that only fails overthe SRs of PLRs.

In some cases, only some of the physical uplinks in the LAG go down. Solong as at least one of the physical uplinks in the LAG remainsfunctional, the gateway machine does not take any action and continuesoperating as normal. Furthermore, in some embodiments, tunnel traffic(within the datacenter) uses a separate LAG. If that entire LAG goesdown, this results in complete tunnel failure, described above insubsection 1.

5. BGP Session Down

In some cases, the BGP session for the SRs may go down (e.g., becausethe MSR process on the gateway machine crashes). When graceful restartis enabled for the BGP process, no failover actions need to be taken solong as the session is reestablished within the timeout set for gracefulrestart. In order to be able to detect when the MSR process (or otherBGP module) has gone down, some embodiments require the process torefresh the status of all BGP sessions periodically, even if the statushas not changed.

On the other hand, if graceful restart is not enabled or the timeout forthe restart expires, the gateway machine of some embodiments sends aphysical connectivity down message to its peer gateway machines thatalso host SRs for the same PLR, in order to indicate that its SR is nolonger functioning. From the perspective of the peer gateway machines,this is the same as if the LAG status is down, in that the SR interfaceson the gateway with the non-functioning BGP session will be taken overby the next-highest ranked SR. In addition, so long as one BGP sessionis functioning, and all physical next hops have the same L3connectivity, then no failover action need be taken.

6. Non-Uniform Routes Among SRs

Failures in the external physical network to which the SRs of a PLRconnect may also affect the SRs. For instance, some external physicalrouters might withdraw a route for a subnet, while other physicalrouters do not. Some embodiments solve this issue locally on the gatewaymachines without involving the central network controllers.

As mentioned, in some embodiments, the SRs have iBGP peering with eachother, and eBGP routes (learned from the external physical routers) aresent over the iBGP sessions without changing the next hop. By referenceto FIG. 10 , any eBGP routes learned by the SR 1015, which have a nexthop of 192.168.2.252 (in the same subnet as the uplink U2), are learnedby the SR 1010 via iBGP. These routes are then installed in the SR 1010with a next hop of 192.168.2.252 because the SR has a dummy interface(U2′) for the actual uplink on the other SR 1015. This same techniquealso happens for route withdrawal scenarios.

7. SR Resurrection

Although SRs may go down for various reasons indicated in the previoussubsections, the SRs will generally be brought back up after a period oftime. This may be indicated at other machines by a BFD session towardsthe particular gateway machine with the SR that had failed coming backup, or by the receipt of a message clearing the physical connectivitydown flag. In some embodiments, the local chassis controller on all ofthe other gateway machines then evaluates whether the local SRs shouldcontinue taking over the remote SRs using the same methodology asdescribed above.

For example, if an IP address currently taken over by a local SR from aremote SR should be given back to the remote SR (i.e., the local SRshould no longer be taking over the IP address), then the local SR stopsanswering ARPs for the IP address. For some embodiments, the localchassis controller removes the IP address from the local SR's southboundinterface. If an IP address should be taken over by a local SR (e.g.,because it has come back up), then it follows the failover proceduredescribed above in subsection 1. In addition, if a local SR isdesignated as standby, and the active SR resumes functioning, then thelocal SR stops advertising prefixes to the external physical routers.Similarly, if a local SR designated as active resumes functioning, italso resumes advertising prefixes.

B. Failure Handling with VM-Based SRs

As noted above, some embodiments use VM (or other data compute nodes) onthe gateway machines to host SRs in a datacenter, rather than (or inaddition to) hosting SRs in DPDK-based datapaths of the gatewaymachines. FIG. 21 illustrates an example physical implementation 2100 ofthree gateway machines 2105-2115 that host the three SRs 2120-2130 for aparticular PLR. Each of the gateway machines includes a MFE, a BGPprocess (e.g., the MSR process described in the above subsection A), anda local control plane, or chassis controller.

The MFEs 2135-2145 on the gateway machines 2105-2115 may be virtualswitches such as OVS, ESX, a different hypervisor-based virtual switch,or other software forwarding elements that can handle distributed L2 andL3 forwarding. As shown, the three MFEs 2135-2145 have a full mesh oftunnels between them, and these three MFEs also have tunnels to MFEslocated at a set of host machines 2150-2155, that host user VMs. Thehost machines 2150-2155 also have local control planes.

The physical implementation 2100 of a network topology with three activeSRs operating as VMs will be used in this subsection to describe variousdifferent failure scenarios. In general, when one of the VMs hosting anSR fails or the tunnels between them fail, the other peer SRs willattempt to take over the failed SR's responsibilities. In someembodiments, the SRs that belong to the same logical router sendheartbeat messages to each other via the transit logical switchperiodically (e.g., by broadcasting a heartbeat message onto the transitlogical switch, which will be delivered to all of the other SRs on thetransit logical switch).

1. Crash of VM Hosting an SR

In some cases, the actual VM that hosts one of the SRs may crash due toany number of reasons. As mentioned above, when the SRs operate inactive-active mode (as in FIG. 21 ), then the management plane assignseach of the VMs a rank for use in failover scenarios. In the case ofFIG. 21 , SR1 on gateway machine 2105 is the highest ranked, SR2 ongateway machine 2110 is the second-highest ranked, and SR3 on gatewaymachine 2115 is the third-highest ranked among the SRs.

FIG. 22 conceptually illustrates the result of one of the VMs crashing.Specifically, this figure illustrates that the VM in which the SR 2125operates on the gateway machine 2110 crashes. As a result, this VM isunable to send out heartbeat messages to the other SRs 2120 and 2130,although the tunnels between the gateway machines are still operational(e.g., for other SRs that operate in other VMs on the gateway machine.In this sense, while the various failure mechanisms affect all of theDPDK-based SRs on a machine (as they are all implemented as VRFs withina datapath), crashes of the VMs for the VM-based SRs only affect thesingle SR operating in that VM, and not the other SRs on the gatewaymachine.

The other SRs 2120 and 2130 detect the failure of the SR 2125 due to themissing heartbeats, and therefore take over responsibility for thefailed SR. Normally, all of the SRs store information for the IPaddresses of their own southbound interfaces as well as the southboundinterfaces of the other SRs. That is, SR 2120 stores information aboutits own interface to the transit logical switch that connects the SRs,as well as the corresponding interface of the SRs 2125 and 2130. The SR2120, however, normally only answers ARP requests for its own interface.

When a SR's VM crashes, as shown in FIG. 22 , the next highest ranked SRthat is still alive is responsible for taking over the failed SRssouthbound interface IP address, as well as any IP addresses the failedSR had previously taken over. For instance, if SR3 2130 had previouslycrashed, then its southbound interface would be taken over by SR2 2125.Thus, FIG. 22 illustrates that the SR 2120 is now acting as both SR1 andSR2. Assuming the logical network forwards northbound packets using ECMPprinciples, the host machines 2150-2155 should route two-thirds of allnorthbound traffic for the logical router to which the SRs 2120-2130belong to the VM on gateway 2105 (e.g., to that VM's MAC address), aspackets forwarded to the IP addresses of both SR1 and SR2 will be routedto that MAC.

In order for the VM on the gateway 2105 to take over the IP address ofSR2 2125, the VM sends GARP messages for this IP address (and, in othercases, all IP addresses that it takes over) to the transit logicalswitch that connects the DR and the SRs 2120-2130. In some embodiments,the VM sends multiple GARP messages in order to better ensure that themessage is received. The MFE 2135 receives these GARP messages, andsends them to the MFE 2145 (for delivery to SR3 2130) as well as to theMFEs at the various hosts 2150-2155 (so that the DR will know to removefrom its ARP cache the old SR2 IP to MAC address mapping).

In the case of two SRs in active-standby mode (e.g., if the SRs belongto a TLR, or a PLR with stateful services configured), then thesouthbound interfaces share the same IP address but with different MACaddresses in some embodiments, as described above. If the standby VMcrashes, then in some embodiments the management plane does not initiatea new standby, on the assumption that the VM will come back up withoutthe active SR's VM also failing. When the active SR's VM fails, however,the standby VM identifies this failure (as no heartbeat messages arereceived), and generates GARP messages so as to remove the mapping ofthe southbound IP address to the crashed SR's MAC address in the ARPtable for the DR in the host machine MFEs (so that these MFEs will routepackets to the new active SR rather than the old active SR). In someembodiments, the tunneling protocol layer (e.g., the VXLAN layer) on thehost machines also learns the MAC:VTEP mapping for the new MAC address.the same IP and MAC addresses are used for the southbound interfaces ofthe active-standby SRs, some embodiments use Reverse ARP (RARP) torefresh the MAC:VTEP mapping at the host machine MFEs (so packets willbe sent over the correct tunnel to the newly active SR).

Lastly, if the standby (now active) VM operates as a SR for a PLR, itbegins route advertisement to the physical external routers. When theBGP process on the gateway machine with the failed SR operates outsideof the VM with the SR, then in some embodiments the local control planeat that gateway machine stops the BGP process from continuing toadvertise routes as well, so that the gateway machine will not attractingress traffic for the failed SR.

2. Complete Tunnel Failure

Complete tunnel failure may occur due to the gateway machine crashing,the MFE on the gateway machine having problems, or due to pNIC orphysical network issues. When complete tunnel failure occurs at aparticular gateway machine, (i) all of the MFEs at host machines withuser VMs or gateway machines lose tunnels to the particular gatewaymachine, (ii) SRs on other gateway machines determine that the SRs onthe particular gateway machine have failed, and (iii) the SRs on theparticular gateway machine determine that the SRs on the other gatewaymachines have failed. In some embodiments, if the particular gatewaymachine no longer receives heartbeat messages on any of the tunnels, thelogic on the particular gateway machine determines that it has lost itstunnel connectivity, not that the other VMs have done so.

FIG. 23 conceptually illustrates the result of complete tunnel failureat the MFE 2145 on the gateway machine 2115 that hosts SR3 2130. Asshown, the MFE 2145 has failed such that the tunnels from this MFE tothe other gateway machines and host machines are down (indicated by thedotted lines). As a result, the other SRs that belong to the same PLR(configured in active-active mode) start a failover process to take overthe southbound interface IP addresses of the failed SR 2130.

In some embodiments, the next-highest ranked SR that is still alive isresponsible for taking over the failed SR's southbound interface IPaddress, as well as any IP addresses the failed SR had previously takenover. Thus, FIG. 23 illustrates that the VM for SR 2125 is now acting asboth SR2 and SR3. Assuming the logical network forwards northboundpackets using ECMP principles, the host machines 2150-2155 should routetwo-thirds of all northbound traffic for the logical router to which theSRs 2120-2130 belong to the VM on gateway 2110 (e.g., to that VMs MACaddress), as packets forwarded to the IP address of both SR2 and SR3will be routed to that MAC.

In order for the VM on the gateway 2110 to take over the IP address ofSR3 2130, the VM sends GARP messages for this IP address (and, in othercases, all IP addresses that it takes over) to the transit logicalswitch that connects the DR and the SR 2120-2130. In some embodiments,the VM sends multiple GARP messages in order to better ensure that themessage is received. The MFE 2140 receives these GARP messages, andsends them to the MFE 2135 (for delivery to SR1 2120) as well as to theMFEs at the various hosts 2150-2155 (so that the DR will know to removefrom its ARP cache the old SR2 IP to MAC address mapping).

In the case of two SRs in active-standby mode (e.g., if the SRs belongto a TLR, or a PLR with stateful services configured), then thesouthbound interfaces share the same IP address but with different MACaddresses in some embodiments. If the tunnels from a gateway machinewith a standby SR fail, then the management plane does not initiate anew standby SR in some embodiments. When the tunnels from a gatewaymachine with an active SR fail, however, the standby VM identifies thisfailure (as no heartbeat messages are received from the active SR), andgenerates GARP messages so as to remove the mapping of the southbound IPaddress to the failed SR's MAC address in the ARP table for the DR inthe host machine MFEs (so that these MFEs will route packets to the newactive SR rather than the old active SR). In some embodiments, thetunneling protocol layer (e.g., the VXLAN layer) on the host machinesalso learns the MAC:VTEP mapping for the new MAC address. Lastly, if thestandby (now active) VM operates as a SR for a PLR, it begins routeadvertisement to the physical external routers. In addition, in someembodiments, the gateway machine with the failed tunnels stops its ownBGP process from continuing to advertise routes.

3. Partial Tunnel Failure

Partial tunnel failure occurs when only some of the tunnels between thegateway machine and other machines in the datacenter go down. This couldbe due to complete failure at one of the machines with a tunnel to theparticular gateway machine (which would result in the loss of onetunnel), due to conditions at the particular gateway machine that resultin some of its tunnels going down, etc. Described here is the case whenconditions at the particular gateway machine result in a subset of itstunnels failing. As a result, (i) some of the MFEs at host machines withuser VMs lose tunnels to the particular gateway machine, (ii) some ofthe other gateway machines lose tunnels to the particular gatewaymachine, and (iii) the particular gateway machine loses tunnels to someother gateway machines.

The MFEs at host machines that lose tunnels to the particular gatewaymachine treat this in the same manner as complete tunnel failure, asfrom the perspective of the host machine this is simply an unreachablegateway. As such, the datapath or chassis controller on the host machineautomatically removes the affected routes that use the SR on theparticular gateway machine as the next hop IP address from the FIB ofthe DR.

As noted, partial tunnel failure can result in various differentscenarios. For instance, in some cases, a gateway machine may bereachable by some of the host machine MFEs, but not by its own peers.Referring to FIG. 13 as an example, the gateway machine 1330 might bereachable by the host machines 1305-1320 but not reachable by gateways1335 and 1340. In this case, the local chassis controller on the gatewaymachine 1330 will take over the SRs of the PLR that are running on bothgateway machines 1335 and 1340. In addition, the gateway machine 1335(or machine 1340, depending on the ranking) will take over the SRrunning on the gateway machine 1330. This results in some of the MFEs(that can reach all of the gateway machines) receiving replies frommultiple gateway machines when the DR running on it sends an ARP requestfor the southbound interface IP address of the SR hosted on the firstgateway machine 1330. So long as the SRs are in an active-activeconfiguration (with no stateful services), this will not create acorrectness problem. However, in the case of an active-standbyconfiguration, this would mean that both of the SRs are now active,which could cause traffic disruption issues.

Partial tunnel failure can also cause problems in active-standby modewhen, at a particular gateway machine, the tunnels to some of the hostmachines go down, but the peer gateway machines remain reachable. Inthis case, because the tunnels between the SRs are functioning, nofailover occurs. In active-active mode, the datapath at the hostmachines (or the local chassis controller) can make the decision toforward traffic over the tunnels that are still up without issue.However, in active-standby mode, if the tunnels to the active SR aredown, then the MFE will send packets to the standby SR, which does notprocess them. Similarly, in both active-active and active-standbyconfigurations, the gateway machine may not be able to pass onsouthbound traffic from physical routers, which is therefore blackholedin some embodiments.

4. vNIC to Physical Router is Down

In some embodiments, each VM on which the SR runs uses a first vNIC toconnect to the MFE for packets sent to and from the physical router(s)(if the SR belongs to a PLR), a second vNIC for sending heartbeatmessages to its peers, and a third vNIC for packets sent to and receivedfrom the logical network. In some embodiments, some or all of thesevNICs may be the same. For instance, the SR might use the same vNIC tosend heartbeat messages and communicate with physical routers, or tosend heartbeat messages and communicate with the logical network.

If the VM loses the first vNIC (with the physical router) for anyreason, in some embodiments the SR stops sending a heartbeat message. Assuch, once its peer VMs that host the other SRs for the PLR detect thatthe heartbeat messages have stopped from the SR, they take failoveractions as described above in subsection 1, as if the VM had crashed. Ifthe VM loses the second vNIC (for heartbeat messages), the peer VMs willdetect that no heartbeat messages are incoming, and take the samefailover actions to take control of the failed SR's IP addresses.Lastly, if the VM loses the third vNIC (for logical network traffic), itindicates the situation in a heartbeat message, and the peers can followthe same failover procedure.

5. BGP Session Down

In some cases, the BGP session for the SRs may go down (e.g., becausethe MSR process on the gateway machine crashes). When graceful restartis enabled for the BGP process, no failover actions need to be taken solong as the session is reestablished within the timeout set for gracefulrestart. In order to be able to detect when the MSR process (or otherBGP module) has gone down, some embodiments require the process torefresh the status of all BGP sessions periodically, even if the statushas not changed.

On the other hand, if graceful restart is not enabled or the timeout forthe restart expires, the gateway machine uses the heartbeat message toindicate that the SR is no longer functioning (e.g., by ceasing theheartbeat messages). From the perspective of the peer SRs, the SR withnon-functioning BGP will be treated as down and the above failoverprocedures apply.

C. Failover Process

FIG. 24 conceptually illustrates a process 2400 performed by a SR incase of failover of a peer SR. In various embodiments, this process maybe performed by either the local control plane operating on the gatewaymachine of the SR (for either a VM or a VRF in a DPDK-based datapath),the SR itself (if implemented as an edge VM), or the datapath (ifimplemented as a VRF in a DPDK-based datapath). That is, the operationsof process 2400 apply to both of the described types of SRs, though theimplementation of the processes may be different for the differenttypes.

As shown, the process 2400 begins by determining (at 2405) that a peerSR has failed. As described in the preceding subsections, a SR mightfail for various reasons, and in different capacities. For example, thetunnel connectivity within the datacenter that enable logical networkcommunication might go down, the ability of the SR to communicate withthe external physical network could become unavailable, the VM thatimplements the SR could crash (if the SR is implemented as such), thedatapath could crash, the entire gateway machine hosting the SR couldcrash, etc. It should be understood that in some cases (e.g., all tunnelconnectivity from the gateway machine going down, the datapath crashing,etc.) all of the SRs on a gateway machine will be considered failed, andtheir various peers will perform the process 2400 or a similar process.

Upon determining that its peer SR has failed, the process 2400 thendetermines (at 2410) whether to take over for the failed peer. Forexample, if the failed peer is the standby SR in an active-standbyconfiguration, then the active SR needs not take any action. Inaddition, for an active-active configuration, only one of the peer SRswill need to take over for a failed SR. As described above, which of theSRs takes over for a particular failed SR is predetermined based on theranks assigned by the management plane at the time of creation of theSRs.

When the SR is not responsible for taking over for the failed SR, theprocess ends. Otherwise, the process identifies (at 2415) the southboundIP addresses owned by the failed peer, for which it is now responsible.These may be different situations in active-active compared toactive-standby mode. Specifically, in active-standby mode, the two SRsshare an IP address on the southbound interface, so the SR will simplytake over acting on its own IP address. In active-active mode, the SRsall have different southbound IP addresses. In this case, the overtakingSR is now responsible for the originally-assigned IP address of thefailed SR, as well as any additional southbound interface IP addressesthat the failed SR had previously taken responsibility for (due tofailure of the other peer SRs).

For each identified southbound IP address, the process 2400 sends (at2420) one or more GARP reply messages to the transit logical switch thatconnects the SRs and the DR of their logical router. The GARP messagesidentify the SR's own southbound MAC address as now associated with thesouthbound IP address or addresses identified at operation 2415. Thisenables the other components on the transit logical switch to cleartheir ARP caches so as to avoid sending packets routed to the identifiedIP address to the failed destination. For the DR, implemented onnumerous gateway and host machines throughout the datacenter, the GARPreply is broadcast to these numerous machines so that the ARP caches onthe various MFEs can be cleared.

The process then determines (at 2425) whether the SR performing theprocess (or the SR on the machine whose local controller chassis isperforming the process) was previously a standby SR of a TLR. It shouldbe understood that the process 2400 is merely conceptual, and thatoperation 2425 is implemented in some embodiments by default on all TLRstandby SRs, and that no specific determination need be made. When thefailed SR was the active SR in an active-standby configuration, thestandby SR is responsible for attracting southbound traffic thatpreviously would have been sent to the failed SR.

Thus, if the SR was formerly a standby SR of a TLR, the process 2400identifies (at 2430) the northbound IP address of the failed peer, whichit shares (as the TLR only is allowed one uplink in some embodiments).The process next sends (at 2430) one or more GARP reply messages to thetransit logical switch that connects the SRs to the DR of a PLR. TheGARP messages identify the SR's own northbound MAC address as nowassociated with the IP address of the uplink configured for the TLR.This enables the DR of the PLR to clear its ARP cache (morespecifically, for the various MFEs that implement this DR across thedatacenter to clear their ARP caches). The process then ends.

If the SR performing the process was not a standby SR of a TLR, theprocess determines (at 2440) whether this SR was previously a standby SRof a PLR. Again, it should be understood that in some embodiments nospecific determination is actually made by the SR or local controllerchassis that performs the process 2400. When this SR was a standby SRfor a PLR, the SR begins advertising (at 2445) prefixes to its externalphysical routers. In the active-active case, the SR would have alreadybeen advertising these prefixes in order to attract ECMP traffic.However, in the active-standby configuration, the standby does notadvertise prefixes, instead only receiving routes from the externalrouters. However, in order to attract southbound traffic, the new active(formerly standby) SR begins advertising prefixes. The process thenends.

VII. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 25 conceptually illustrates an electronic system 2500 with whichsome embodiments of the invention are implemented. The electronic system2500 can be used to execute any of the control, virtualization, oroperating system applications described above. The electronic system2500 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system2500 includes a bus 2505, processing unit(s) 2510, a system memory 2525,a read-only memory 2530, a permanent storage device 2535, input devices2540, and output devices 2545.

The bus 2505 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 2500. For instance, the bus 2505 communicativelyconnects the processing unit(s) 2510 with the read-only memory 2530, thesystem memory 2525, and the permanent storage device 2535.

From these various memory units, the processing unit(s) 2510 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 2530 stores static data and instructions thatare needed by the processing unit(s) 2510 and other modules of theelectronic system. The permanent storage device 2535, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system2500 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 2535.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2535, the system memory 2525 is a read-and-write memorydevice. However, unlike storage device 2535, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2525, the permanent storage device 2535, and/or theread-only memory 2530. From these various memory units, the processingunit(s) 2510 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2505 also connects to the input and output devices 2540 and2545. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 2540 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2545 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 25 , bus 2505 also couples electronic system2500 to a network 2565 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 2500 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

This specification refers throughout to computational and networkenvironments that include virtual machines (VMs). However, virtualmachines are merely one example of data compute nodes (DCNs) or datacompute end nodes, also referred to as addressable nodes. DCNs mayinclude non-virtualized physical hosts, virtual machines, containersthat run on top of a host operating system without the need for ahypervisor or separate operating system, and hypervisor kernel networkinterface modules.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. In someembodiments, the host operating system uses name spaces to isolate thecontainers from each other and therefore provides operating-system levelsegregation of the different groups of applications that operate withindifferent containers. This segregation is akin to the VM segregationthat is offered in hypervisor-virtualized environments that virtualizesystem hardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface modules, in some embodiments, is anon-VM DCN that includes a network stack with a hypervisor kernelnetwork interface and receive/transmit threads. One example of ahypervisor kernel network interface module is the vmknic module that ispart of the ESXi™ hypervisor of VMware, Inc.

It should be understood that while the specification refers to VMs, theexamples given could be any type of DCNs, including physical hosts, VMs,non-VM containers, and hypervisor kernel network interface modules. Infact, the example networks could include combinations of different typesof DCNs in some embodiments.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 11, 12, and 24 ) conceptually illustrate processes. Thespecific operations of these processes may not be performed in the exactorder shown and described. The specific operations may not be performedin one continuous series of operations, and different specificoperations may be performed in different embodiments. Furthermore, theprocess could be implemented using several sub-processes, or as part ofa larger macro process. Thus, one of ordinary skill in the art wouldunderstand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims.

1-22. (canceled)
 23. A method for processing data messages between alogical first network implemented in a datacenter and a second networkexternal to the logical network, the method comprising: at an edge nodethat operates at an edge of the datacenter: from the external secondnetwork, receiving a data message that is addressed to a logical networkdata compute node (DCN); applying, to the data message, configurationfor (i) a centralized routing component of a logical router that is oneof a plurality of centralized routing components of the logical routerthat are each implemented by different edge nodes, (ii) a distributedrouting component of the logical router that is implemented by each ofthe edge nodes and a plurality of additional forwarding elements, and(iii) a logical switch of the logical network; and based on theconfiguration for the logical switch, forwarding the data messagethrough the datacenter to the DCN.
 24. The method of claim 23, whereinthe logical switch is logically connected to (i) the DCN and (ii) thedistributed routing component of the logical router.
 25. The method ofclaim 24, wherein forwarding the data message through the datacentercomprises tunneling the data message to a host computer on which the DCNexecutes.
 26. The method of claim 23 further comprising applyingconfiguration for a transit logical switch that logically connects theplurality of centralized routing components of the logical router to thedistributed routing component of the logical router.
 27. The method ofclaim 23, wherein: the logical router is a first logical router; themethod further comprises applying configuration for a distributedrouting component of a second logical router after applying theconfiguration for the centralized and distributed routing components ofthe first logical router; and the logical switch is logically connectedto (i) the DCN and (ii) the distributed routing component of the secondlogical router.
 28. The method of claim 27, wherein the first logicalrouter is a provider-configured logical router that provides connectionto the external second network and the second logical router is atenant-configured logical router that enables a tenant to configurespecific services and policies.
 29. The method of claim 23, wherein: thelogical router is a first logical router; the method further comprisesapplying configuration for (i) a centralized routing component of asecond logical router that is one of a plurality of centralized routingcomponents of the second logical router that are each implemented bydifferent edge nodes and (ii) a distributed routing component of thesecond logical router; and the logical switch is logically connected to(i) the DCN and (ii) the distributed routing component of the secondlogical router.
 30. The method of claim 29 further comprising applyingconfiguration for (i) a first transit logical switch that logicallyconnects the plurality of centralized routing components of the firstlogical router to the distributed routing component of the first logicalrouter, (ii) a second transit logical switch that logically connects thedistributed routing component of the first logical router to theplurality of centralized routing components of the second logicalrouter, and (iii) a third transit logical switch that logically connectsthe plurality of centralized routing components of the second logicalrouter to the distributed routing component of the second logicalrouter.
 31. The method of claim 23, wherein: the logical router is afirst logical router and the edge node is a first edge node; the logicalswitch is logically connected to the distributed routing component ofthe first logical router and a plurality of centralized routingcomponents of a second logical router; forwarding the data messagethrough the datacenter to the DCN comprises tunneling the data messageto a second edge node that implements one of the centralized routingcomponents of the second logical router; and the second edge nodeapplies configuration for the (i) centralized routing component of thesecond logical router and (ii) a distributed routing component of thesecond logical router and tunnels the data message to a host computer onwhich the DCN executes.
 32. The method of claim 31, wherein both thefirst edge node and the second edge node apply configuration for thelogical switch, wherein the second edge node also applies configurationfor a second logical switch that is logically connected to the DCN andthe distributed routing component of the second logical router.
 33. Anon-transitory machine-readable medium storing a program which whenexecuted by at least one processing unit of an edge node, that operatesat an edge of a datacenter, processes data messages between a logicalfirst network implemented in a datacenter and a second network externalto the logical network, the program comprising sets of instructions for:from the external second network, receiving a data message that isaddressed to a logical network data compute node (DCN); applying, to thedata message, configuration for (i) a centralized routing component of alogical router that is one of a plurality of centralized routingcomponents of the logical router that are each implemented by differentedge nodes, (ii) a distributed routing component of the logical routerthat is implemented by each of the edge nodes and a plurality ofadditional forwarding elements, and (iii) a logical switch of thelogical network; and based on the configuration for the logical switch,forwarding the data message through the datacenter to the DCN.
 34. Thenon-transitory machine-readable medium of claim 33, wherein the logicalswitch is logically connected to (i) the DCN and (ii) the distributedrouting component of the logical router.
 35. The non-transitorymachine-readable medium of claim 34, wherein the set of instructions forforwarding the data message through the datacenter comprises a set ofinstructions for tunneling the data message to a host computer on whichthe DCN executes.
 36. The non-transitory machine-readable medium ofclaim 33, wherein the program further comprises a set of instructionsfor applying configuration for a transit logical switch that logicallyconnects the plurality of centralized routing components of the logicalrouter to the distributed routing component of the logical router. 37.The non-transitory machine-readable medium of claim 33, wherein: thelogical router is a first logical router; the program further comprisesa set of instructions for applying configuration for a distributedrouting component of a second logical router after applying theconfiguration for the centralized and distributed routing components ofthe first logical router; and the logical switch is logically connectedto (i) the DCN and (ii) the distributed routing component of the secondlogical router.
 38. The non-transitory machine-readable medium of claim37, wherein the first logical router is a provider-configured logicalrouter that provides connection to the external second network and thesecond logical router is a tenant-configured logical router that enablesa tenant to configure specific services and policies.
 39. Thenon-transitory machine-readable medium of claim 33, wherein: the logicalrouter is a first logical router; the program further comprises sets ofinstructions for applying configuration for (i) a centralized routingcomponent of a second logical router that is one of a plurality ofcentralized routing components of the second logical router that areeach implemented by different edge nodes and (ii) a distributed routingcomponent of the second logical router; and the logical switch islogically connected to (i) the DCN and (ii) the distributed routingcomponent of the second logical router.
 40. The non-transitorymachine-readable medium of claim 39, wherein the program furthercomprises a set of instructions for applying configuration for (i) afirst transit logical switch that logically connects the plurality ofcentralized routing components of the first logical router to thedistributed routing component of the first logical router, (ii) a secondtransit logical switch that logically connects the distributed routingcomponent of the first logical router to the plurality of centralizedrouting components of the second logical router, and (iii) a thirdtransit logical switch that logically connects the plurality ofcentralized routing components of the second logical router to thedistributed routing component of the second logical router.
 41. Thenon-transitory machine-readable medium of claim 33, wherein: the logicalrouter is a first logical router and the edge node is a first edge node;the logical switch is logically connected to the distributed routingcomponent of the first logical router and a plurality of centralizedrouting components of a second logical router; the set of instructionsfor forwarding the data message through the datacenter to the DCNcomprises a set of instructions for tunneling the data message to asecond edge node that implements one of the centralized routingcomponents of the second logical router; and the second edge nodeapplies configuration for the (i) centralized routing component of thesecond logical router and (ii) a distributed routing component of thesecond logical router and tunnels the data message to a host computer onwhich the DCN executes.
 42. The non-transitory machine-readable mediumof claim 41, wherein both the first edge node and the second edge nodeapply configuration for the logical switch, wherein the second edge nodealso applies configuration for a second logical switch that is logicallyconnected to the DCN and the distributed routing component of the secondlogical router.